[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489721154 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2854280K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003337] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010096] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013092] *** VALIDATE selinux *** [ 0.021785] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025664] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027025] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029045] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.032044] *** VALIDATE proc *** [ 0.033226] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.042900] debug: unmapping init [mem 0xffffffffa0859000-0xffffffffa0860fff] [ 0.044180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047027] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053305] rcu: Hierarchical SRCU implementation. [ 0.055474] smp: Bringing up secondary CPUs ... [ 0.056618] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060453] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139387] node 0 deferred pages initialised in 74ms [ 0.143308] devtmpfs: initialized [ 0.144178] x86/mm: Memory block size: 128MB [ 0.147997] gcov: version magic: 0x41383552 [ 0.150238] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152277] pinctrl core: initialized pinctrl subsystem [ 0.153191] [ 0.153775] ************************************************************* [ 0.154023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155015] ** ** [ 0.156014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157010] ** ** [ 0.158015] ** This means that this kernel is built to expose internal ** [ 0.159015] ** IOMMU data structures, which may compromise security on ** [ 0.160020] ** your system. ** [ 0.161016] ** ** [ 0.162016] ** If you see this message and you are not debugging the ** [ 0.163014] ** kernel, report this immediately to your vendor! ** [ 0.164014] ** ** [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166015] ************************************************************* [ 0.167636] NET: Registered protocol family 16 [ 0.168680] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172043] cpuidle: using governor menu [ 0.173840] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176463] PCI: Using configuration type 1 for base access [ 0.179126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192064] cryptd: max_cpu_qlen set to 1000 [ 0.195292] ACPI: Added _OSI(Module Device) [ 0.196018] ACPI: Added _OSI(Processor Device) [ 0.197020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202218] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206327] ACPI: Interpreter enabled [ 0.207089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208019] ACPI: Using IOAPIC for interrupt routing [ 0.209172] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219477] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220144] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.222116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224378] acpiphp: Slot [2] registered [ 0.225253] acpiphp: Slot [5] registered [ 0.226179] acpiphp: Slot [6] registered [ 0.227191] acpiphp: Slot [7] registered [ 0.228139] acpiphp: Slot [8] registered [ 0.229147] acpiphp: Slot [9] registered [ 0.231125] acpiphp: Slot [10] registered [ 0.233134] acpiphp: Slot [3] registered [ 0.235116] acpiphp: Slot [4] registered [ 0.237237] acpiphp: Slot [11] registered [ 0.238206] acpiphp: Slot [12] registered [ 0.240105] acpiphp: Slot [13] registered [ 0.242129] acpiphp: Slot [14] registered [ 0.244155] acpiphp: Slot [15] registered [ 0.245238] acpiphp: Slot [16] registered [ 0.247225] acpiphp: Slot [17] registered [ 0.249106] acpiphp: Slot [18] registered [ 0.250106] acpiphp: Slot [19] registered [ 0.252103] acpiphp: Slot [20] registered [ 0.253087] acpiphp: Slot [21] registered [ 0.255191] acpiphp: Slot [22] registered [ 0.257091] acpiphp: Slot [23] registered [ 0.258110] acpiphp: Slot [24] registered [ 0.260111] acpiphp: Slot [25] registered [ 0.262101] acpiphp: Slot [26] registered [ 0.263244] acpiphp: Slot [27] registered [ 0.265164] acpiphp: Slot [28] registered [ 0.267109] acpiphp: Slot [29] registered [ 0.269109] acpiphp: Slot [30] registered [ 0.270091] acpiphp: Slot [31] registered [ 0.271084] PCI host bridge to bus 0000:00 [ 0.273019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294090] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303783] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.308075] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.310023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315587] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.322046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324673] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.361015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.400496] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408041] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.415017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.433017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.444983] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.452017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.459017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.475018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.485835] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.494024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.526031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.541013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.549013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.586563] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.595020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.619020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.631196] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634571] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637390] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.642282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647116] iommu: Default domain type: Passthrough [ 0.648554] SCSI subsystem initialized [ 0.649240] ACPI: bus type USB registered [ 0.650000] usbcore: registered new interface driver usbfs [ 0.654099] usbcore: registered new interface driver hub [ 0.656091] usbcore: registered new device driver usb [ 0.659193] pps_core: LinuxPPS API ver. 1 registered [ 0.661012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665301] PTP clock support registered [ 0.668122] EDAC MC: Ver: 3.0.0 [ 0.670188] PCI: Using ACPI for IRQ routing [ 0.673325] NetLabel: Initializing [ 0.674013] NetLabel: domain hash size = 128 [ 0.676017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.678085] NetLabel: unlabeled traffic allowed by default [ 0.680166] vgaarb: loaded [ 0.681179] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.683016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.690377] clocksource: Switched to clocksource kvm-clock [ 0.797880] VFS: Disk quotas dquot_6.6.0 [ 0.800469] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.803128] *** VALIDATE ramfs *** [ 0.804390] *** VALIDATE hugetlbfs *** [ 0.806054] pnp: PnP ACPI init [ 0.808626] pnp: PnP ACPI: found 6 devices [ 0.828038] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.831133] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.833463] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.835864] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.838364] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840903] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.844657] NET: Registered protocol family 2 [ 0.848513] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.854879] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.859299] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.865132] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.870453] TCP: Hash tables configured (established 65536 bind 65536) [ 0.873802] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880972] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884824] NET: Registered protocol family 1 [ 0.888725] RPC: Registered named UNIX socket transport module. [ 0.891853] RPC: Registered udp transport module. [ 0.894829] RPC: Registered tcp transport module. [ 0.896929] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.899598] NET: Registered protocol family 44 [ 0.901539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904099] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.906947] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.909863] PCI: CLS 0 bytes, default 64 [ 0.911647] Unpacking initramfs... [ 2.288599] debug: unmapping init [mem 0xffff991afcc54000-0xffff991afffbffff] [ 2.292284] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.294221] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.296389] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.768222] Initialise system trusted keyrings [ 2.769945] Key type blacklist registered [ 2.771863] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.780238] zbud: loaded [ 2.782348] *** VALIDATE nfs *** [ 2.783678] *** VALIDATE nfs4 *** [ 2.785477] pstore: using deflate compression [ 2.788702] Platform Keyring initialized [ 2.884398] NET: Registered protocol family 38 [ 2.886422] Key type asymmetric registered [ 2.889061] Asymmetric key parser 'x509' registered [ 2.891463] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.894742] io scheduler mq-deadline registered [ 2.896806] io scheduler kyber registered [ 2.898942] io scheduler bfq registered [ 2.901115] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.904802] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.907852] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.911337] ACPI: Power Button [PWRF] [ 2.918702] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.925065] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.936343] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.942957] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.956743] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.985837] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.013753] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.019577] Non-volatile memory driver v1.3 [ 3.021586] Linux agpgart interface v0.103 [ 3.062520] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.068083] vda: detected capacity change from 0 to 76677120 [ 3.082313] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.085609] vdb: detected capacity change from 0 to 1073741824 [ 3.099613] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.102053] vdc: detected capacity change from 0 to 2621440000 [ 3.114334] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.117517] vdd: detected capacity change from 0 to 2621440000 [ 3.132061] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.135060] vde: detected capacity change from 0 to 4294967296 [ 3.149819] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.153409] vdf: detected capacity change from 0 to 4294967296 [ 3.160465] libphy: Fixed MDIO Bus: probed [ 3.165436] usbcore: registered new interface driver usbserial_generic [ 3.167878] usbserial: USB Serial support registered for generic [ 3.169805] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.173356] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.174935] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.177475] mousedev: PS/2 mouse device common for all mice [ 3.180559] rtc_cmos 00:05: RTC can wake from S4 [ 3.183568] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.183773] rtc_cmos 00:05: registered as rtc0 [ 3.190132] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.191777] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.193452] intel_pstate: CPU model not supported [ 3.199297] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.205447] hid: raw HID events driver (C) Jiri Kosina [ 3.207648] usbcore: registered new interface driver usbhid [ 3.209258] usbhid: USB HID core driver [ 3.210421] drop_monitor: Initializing network drop monitor service [ 3.212068] Initializing XFRM netlink socket [ 3.213335] NET: Registered protocol family 10 [ 3.216361] Segment Routing with IPv6 [ 3.217526] NET: Registered protocol family 17 [ 3.219382] mpls_gso: MPLS GSO support [ 3.223563] RAS: Correctable Errors collector initialized. [ 3.225614] AVX version of gcm_enc/dec engaged. [ 3.227751] AES CTR mode by8 optimization enabled [ 3.293926] sched_clock: Marking stable (3293897419, 0)->(4270488633, -976591214) [ 3.297954] registered taskstats version 1 [ 3.300373] Loading compiled-in X.509 certificates [ 3.302640] zswap: loaded using pool lzo/zbud [ 3.325556] Key type big_key registered [ 3.335789] Key type encrypted registered [ 3.337377] ima: No TPM chip found, activating TPM-bypass! [ 3.339557] ima: Allocated hash algorithm: sha1 [ 3.341277] ima: No architecture policies found [ 3.342897] evm: Initialising EVM extended attributes: [ 3.344405] evm: security.selinux [ 3.345643] evm: security.ima [ 3.346812] evm: security.capability [ 3.347746] evm: HMAC attrs: 0x1 [ 3.350321] rtc_cmos 00:05: setting system clock to 2026-09-03 12:48:52 UTC (1788439732) [ 3.355523] debug: unmapping init [mem 0xffffffffa1803000-0xffffffffa19fffff] [ 3.359184] debug: unmapping init [mem 0xffffffffa0582000-0xffffffffa0858fff] [ 3.368476] Write protecting the kernel read-only data: 28672k [ 3.372185] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 3.375472] debug: unmapping init [mem 0xffffffff9f514000-0xffffffff9f5fffff] [ 3.408606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.416255] systemd[1]: Detected virtualization kvm. [ 3.418455] systemd[1]: Detected architecture x86-64. [ 3.420593] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.446099] systemd[1]: No hostname configured. [ 3.448038] systemd[1]: Set hostname to . [ 3.450389] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452865] systemd[1]: Initializing machine ID from random generator. [ 3.498459] random: ln: uninitialized urandom read (6 bytes read) [ 3.597027] random: systemd: uninitialized urandom read (16 bytes read) [ 3.599676] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.606187] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.616492] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.240393] device-mapper: uevent: version 1.0.3 [ 4.242761] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.908801] virtio_net virtio0 ens2: renamed from eth0 [ 4.923383] scsi host0: ata_piix [ 4.928590] random: fast init done [ 4.948347] scsi host1: ata_piix [ 4.950644] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.956488] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.611074] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.876601] random: crng init done [ 9.877737] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.196522] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.335582] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.581996] SELinux: Disabled at runtime. [ 11.639931] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.650346] systemd[1]: Detected virtualization kvm. [ 11.652417] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.130982] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.134664] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.138402] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.141791] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.145041] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.153037] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.156707] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.242799] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.638644] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.949877] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.953346] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.037090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.059958] EDAC sbridge: Ver: 1.1.2 [ 14.786307] Key type dns_resolver registered [ 15.108301] NFS: Registering the id_resolver key type [ 15.112535] Key type id_resolver registered [ 15.113798] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-server login: [ 59.824675] libcfs: loading out-of-tree module taints kernel. [ 59.912754] Key type ._llcrypt registered [ 59.919915] Key type .llcrypt registered [ 60.077898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 77.646009] hrtimer: interrupt took 3998915 ns [ 80.742993] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 82.526706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.546656] alg: No test for adler32 (adler32-zlib) [ 83.973247] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 84.822632] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 86.615145] Key type lgssc registered [ 88.579832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.561959] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 150.200561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 162.858312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.897085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.218456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.280572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.408604] Lustre: lustre-MDT0000: new disk, initializing [ 164.536501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.569727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.300568] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.930470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.074832] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.131561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.137566] Lustre: Skipped 1 previous similar message [ 182.303511] Lustre: lustre-MDT0001: new disk, initializing [ 182.374553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.429050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.438840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.732484] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.390210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.405890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.698876] Lustre: lustre-OST0000: new disk, initializing [ 200.701841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.706552] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.770987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.228644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.245113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.352444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.336125] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.136461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.228463] Lustre: lustre-OST0001: new disk, initializing [ 219.230710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.237681] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.269188] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.837203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.849571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.927490] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.427670] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.097736] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.170758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.447085] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.145127] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 263.635265] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 267.525073] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 270.012328] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 271.781635] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:53:19 EDT 2026 [ 289.609736] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 291.102454] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.243215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.256914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.279660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.800557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.801229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.809567] Lustre: Skipped 2 previous similar messages [ 312.819151] Lustre: Skipped 3 previous similar messages [ 314.280357] Lustre: server umount lustre-MDT0000 complete [ 318.388971] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440047 with bad export cookie 9551478666918814970 [ 318.402108] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.416552] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 323.054587] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.058264] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.063168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.068494] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 323.103764] Lustre: Skipped 1 previous similar message [ 324.716674] Lustre: server umount lustre-MDT0001 complete [ 328.867462] Lustre: server umount lustre-OST0000 complete [ 332.748748] Lustre: server umount lustre-OST0001 complete [ 339.154357] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 348.940557] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 382.883045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.131451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.171825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 383.251561] Lustre: lustre-MDT0000: new disk, initializing [ 383.347513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.377520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.090913] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 398.290251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.530094] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.610103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 398.618042] Lustre: Skipped 1 previous similar message [ 398.699237] Lustre: lustre-MDT0001: new disk, initializing [ 398.905569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.950695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.963375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.689347] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 414.290799] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 421.433759] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.154526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.509129] Lustre: lustre-OST0000: new disk, initializing [ 429.514787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.536247] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.680435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.486390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.507290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.639480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.128196] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 445.462303] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 450.406948] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.570593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.448259] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.648374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.304832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.314651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.343060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.356880] Lustre: Skipped 2 previous similar messages [ 467.424665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.031530] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 472.150375] Lustre: server umount lustre-OST0000 complete [ 477.667273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.669382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.681107] Lustre: Skipped 2 previous similar messages [ 477.688904] Lustre: Skipped 1 previous similar message [ 482.339401] Lustre: server umount lustre-MDT0000 complete [ 482.787468] LustreError: 15930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.829590] LustreError: 15930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 486.496175] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440215 with bad export cookie 9551478666918816503 [ 486.506937] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.510059] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 486.962845] Lustre: server umount lustre-MDT0001 complete [ 490.615978] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:56:58 (1788440218) [ 497.301210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.851142] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.924540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.609919] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 503.269694] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.398916] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.050973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.256729] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 524.783711] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.009149] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 537.878916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.892948] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 552.300557] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 563.170690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.173818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.185811] LustreError: Skipped 1 previous similar message [ 563.191711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.200211] Lustre: Skipped 1 previous similar message [ 563.222520] Lustre: Skipped 1 previous similar message [ 564.809647] Lustre: server umount lustre-OST0000 complete [ 568.804753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.071128] Lustre: server umount lustre-MDT0000 complete [ 578.541391] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.567535] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 578.662601] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440307 with bad export cookie 9551478666918817385 [ 578.667657] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.678828] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 579.127267] Lustre: server umount lustre-MDT0001 complete [ 591.062523] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 594.502647] Key type lgssc unregistered [ 594.862597] LNet: 22843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.871212] LNetError: 22843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.891798] LNet: Removed LNI 192.168.203.137@tcp [ 596.300896] Key type .llcrypt unregistered [ 596.305351] Key type ._llcrypt unregistered [ 621.640890] Key type ._llcrypt registered [ 621.645239] Key type .llcrypt registered [ 621.763869] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 622.694890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.764339] alg: No test for adler32 (adler32-zlib) [ 623.969359] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 624.294932] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 625.999758] Key type lgssc registered [ 627.335849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 635.273265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 635.292825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.823960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.608548] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 646.114152] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.129261] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 650.333141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.829995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 655.394069] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 662.847322] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 669.329645] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 677.250352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.644887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.809747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 683.319715] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 690.501293] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 707.552807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.569403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.596000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 708.298582] Lustre: server umount lustre-OST0000 complete [ 712.160587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 712.168678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.181064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.744589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.745728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.278973] Lustre: server umount lustre-MDT0000 complete [ 720.873764] LustreError: 25661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.903179] LustreError: 25661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 721.923051] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440451 with bad export cookie 490420648106895046 [ 721.947089] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.951125] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 732.441415] Lustre: server umount lustre-MDT0001 complete [ 743.666934] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 746.571795] Key type lgssc unregistered [ 746.887664] LNet: 28254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.899976] LNetError: 28254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.916241] LNet: Removed LNI 192.168.203.137@tcp [ 747.932369] Key type .llcrypt unregistered [ 747.935899] Key type ._llcrypt unregistered [ 771.402213] Key type ._llcrypt registered [ 771.404313] Key type .llcrypt registered [ 771.477701] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 772.108688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.204178] alg: No test for adler32 (adler32-zlib) [ 773.299077] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 773.642659] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 775.393744] Key type lgssc registered [ 776.792548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 785.416973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 785.453537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.099116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.889518] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 797.668956] LustreError: 29662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.696966] LustreError: 29662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 802.793082] LustreError: 29663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.320510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.777948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 809.209596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 816.779805] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 822.026356] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 834.506045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.861897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.658517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 849.236936] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 857.400467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 867.809291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.851215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 867.862874] Lustre: Skipped 1 previous similar message [ 872.252701] Lustre: server umount lustre-OST0000 complete [ 876.512549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 876.522656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.534183] Lustre: Skipped 1 previous similar message [ 876.547039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.052639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.074884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.169148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.175973] Lustre: Skipped 2 previous similar messages [ 888.289134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.298737] Lustre: Skipped 1 previous similar message [ 890.335233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 890.611835] Lustre: server umount lustre-MDT0000 complete [ 893.410379] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.426632] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 894.548112] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440623 with bad export cookie 15986497835496151858 [ 894.554187] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.562731] LustreError: 29643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 895.207311] Lustre: server umount lustre-MDT0001 complete [ 907.012685] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 910.487896] Key type lgssc unregistered [ 910.872381] LNet: 33566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.882900] LNetError: 33566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.905052] LNet: Removed LNI 192.168.203.137@tcp [ 912.087240] Key type .llcrypt unregistered [ 912.089073] Key type ._llcrypt unregistered [ 938.437427] Key type ._llcrypt registered [ 938.438737] Key type .llcrypt registered [ 938.571397] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 939.578728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 939.805535] alg: No test for adler32 (adler32-zlib) [ 940.873571] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 941.203934] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 942.927161] Key type lgssc registered [ 944.176854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.465490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.497711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.095471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.030910] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 964.576845] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.597762] LustreError: 34992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 968.056691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.350926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.833260] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 980.170336] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.618772] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.679889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.978194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.413572] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 999.158880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1003.643355] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.726488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1012.739714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.769186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.732483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.748552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.770017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.581421] Lustre: server umount lustre-OST0000 complete [ 1024.483849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1024.504128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.523173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.538773] Lustre: Skipped 2 previous similar messages [ 1029.534713] Lustre: server umount lustre-MDT0000 complete [ 1030.119296] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.151742] LustreError: 34989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1032.957132] LustreError: 34973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440762 with bad export cookie 14806887607388118676 [ 1032.960939] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1032.963112] LustreError: 34973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1033.199684] Lustre: server umount lustre-MDT0001 complete [ 1043.679074] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1046.813611] Key type lgssc unregistered [ 1047.440692] LNet: 38522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.457972] LNetError: 38522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.515643] LNet: Removed LNI 192.168.203.137@tcp [ 1049.000741] Key type .llcrypt unregistered [ 1049.017214] Key type ._llcrypt unregistered [ 1073.806545] Key type ._llcrypt registered [ 1073.810955] Key type .llcrypt registered [ 1073.941609] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1075.338625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.427748] alg: No test for adler32 (adler32-zlib) [ 1076.481172] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1076.738274] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1078.439326] Key type lgssc registered [ 1079.175939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1086.640741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1086.654759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.258149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.630097] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1098.731130] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.757225] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1103.308075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.669690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.553683] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1115.442447] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1121.329882] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1127.775548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.143779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.299094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1134.542658] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1141.190296] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1148.898350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.907634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1148.923431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.940300] Lustre: Skipped 1 previous similar message [ 1154.027697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.036159] Lustre: Skipped 2 previous similar messages [ 1159.136887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.144637] Lustre: Skipped 2 previous similar messages [ 1162.208283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1162.336426] Lustre: server umount lustre-OST0000 complete [ 1181.609671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.637720] Lustre: Skipped 1 previous similar message [ 1181.646929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.663398] Lustre: Skipped 2 previous similar messages [ 1186.797521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.821887] Lustre: Skipped 1 previous similar message [ 1188.208956] Lustre: server umount lustre-MDT0000 complete [ 1188.897085] LustreError: 39927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.938430] LustreError: 39927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1191.874450] LustreError: 39912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440921 with bad export cookie 17396613064339918368 [ 1191.892932] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.239120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1206.518660] Lustre: server umount lustre-MDT0001 complete [ 1216.072508] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1219.711339] Key type lgssc unregistered [ 1220.264757] LNet: 43641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.272804] LNetError: 43641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.312660] LNet: Removed LNI 192.168.203.137@tcp [ 1221.447922] Key type .llcrypt unregistered [ 1221.450109] Key type ._llcrypt unregistered [ 1247.247946] Key type ._llcrypt registered [ 1247.250350] Key type .llcrypt registered [ 1247.394613] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1248.814751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.862581] alg: No test for adler32 (adler32-zlib) [ 1250.040220] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1250.400596] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1252.119166] Key type lgssc registered [ 1253.497580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.657660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.679833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.218569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.490130] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1273.837237] LustreError: 45051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.856559] LustreError: 45051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1275.369599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.784398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.300441] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1287.311307] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1292.517747] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1298.995076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.418156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.509836] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1308.676280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1313.139164] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1318.897329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.922367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.185969] Lustre: server umount lustre-MDT0000 complete [ 1319.979675] LustreError: 45045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.889821] LustreError: 45030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441052 with bad export cookie 8515920083559243649 [ 1322.897983] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.904224] LustreError: 45030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1324.000506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.002693] LustreError: 45047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.008864] Lustre: Skipped 2 previous similar messages [ 1324.010779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1324.029418] Lustre: Skipped 2 previous similar messages [ 1325.097879] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 1328.034371] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.036124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1328.068652] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1329.298735] Lustre: server umount lustre-MDT0001 complete [ 1350.586380] Lustre: server umount lustre-OST0000 complete [ 1366.489898] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1368.879530] Key type lgssc unregistered [ 1369.152348] LNet: 48909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.160362] LNetError: 48909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.176832] LNet: Removed LNI 192.168.203.137@tcp [ 1369.900168] Key type .llcrypt unregistered [ 1369.902538] Key type ._llcrypt unregistered [ 1392.038488] Key type ._llcrypt registered [ 1392.041358] Key type .llcrypt registered [ 1392.114390] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1393.208568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.274741] alg: No test for adler32 (adler32-zlib) [ 1394.349254] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1394.618069] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1396.263179] Key type lgssc registered [ 1397.996587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.353617] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.431832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.031538] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1429.151302] LustreError: 50273:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 1455.839188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.057594] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1472.976545] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1640.235968] Lustre: server umount lustre-OST0000 complete [ 1655.379220] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1657.928692] Key type lgssc unregistered [ 1658.267837] LNet: 51946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.278748] LNetError: 51946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.297055] LNet: Removed LNI 192.168.203.137@tcp [ 1659.141909] Key type .llcrypt unregistered [ 1659.150897] Key type ._llcrypt unregistered [ 1681.459318] Key type ._llcrypt registered [ 1681.465324] Key type .llcrypt registered [ 1681.632488] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1682.354492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.388685] alg: No test for adler32 (adler32-zlib) [ 1683.501766] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1683.759097] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1685.415184] Key type lgssc registered [ 1686.435129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1694.117622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1694.141312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.631692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.755232] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1704.936339] LustreError: 53358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.956825] LustreError: 53358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1710.074800] LustreError: 53357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.358391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.840109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.519263] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1723.737983] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.033971] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.174067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.519276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.655066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1741.138891] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1748.194983] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1751.011170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.012480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.037448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.134762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.150329] Lustre: Skipped 2 previous similar messages [ 1756.694943] Lustre: server umount lustre-OST0000 complete [ 1761.258550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.283744] Lustre: Skipped 1 previous similar message [ 1761.294639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.307770] Lustre: Skipped 1 previous similar message [ 1766.374255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.387180] Lustre: Skipped 1 previous similar message [ 1766.782730] Lustre: server umount lustre-MDT0000 complete [ 1770.776073] LustreError: 55199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441499 with bad export cookie 3753396879920125725 [ 1770.792376] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.797725] LustreError: 55199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1771.187955] Lustre: server umount lustre-MDT0001 complete [ 1783.373846] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 1786.422251] Key type lgssc unregistered [ 1786.756665] LNet: 56888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.763730] LNetError: 56888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.778799] LNet: Removed LNI 192.168.203.137@tcp [ 1787.690927] Key type .llcrypt unregistered [ 1787.694270] Key type ._llcrypt unregistered [ 1811.345942] Key type ._llcrypt registered [ 1811.351268] Key type .llcrypt registered [ 1811.471157] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 1812.665183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.797026] alg: No test for adler32 (adler32-zlib) [ 1813.961790] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1814.322930] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 1816.007451] Key type lgssc registered [ 1817.563949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.499906] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.518719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.183421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1846.303636] LustreError: 58272:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 1872.991293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.015220] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1888.164438] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1895.733931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.192789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.239181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1900.103175] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1901.537228] LustreError: 58298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.562537] LustreError: 58298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1906.658947] LustreError: 58297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.685043] LustreError: 59309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.719271] LustreError: 59309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1908.323376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.832255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.436413] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 1925.493480] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1932.377293] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1934.308166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.309534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.312642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.328627] Lustre: Skipped 1 previous similar message [ 1939.431764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.434783] Lustre: Skipped 1 previous similar message [ 1940.501506] Lustre: server umount lustre-OST0000 complete [ 1963.873254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.890342] Lustre: Skipped 1 previous similar message [ 1963.896422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.921100] Lustre: Skipped 1 previous similar message [ 1969.121026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.125872] Lustre: Skipped 1 previous similar message [ 1974.240737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.248737] Lustre: Skipped 1 previous similar message [ 1975.263139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.495305] Lustre: server umount lustre-MDT0000 complete [ 1979.360574] LustreError: 61015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.382986] LustreError: 61015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1981.183316] LustreError: 58279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441710 with bad export cookie 2849888977009835772 [ 1981.196236] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.786967] Lustre: server umount lustre-MDT0001 complete [ 1998.827170] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2001.753924] Key type lgssc unregistered [ 2002.043342] LNet: 62020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.061422] LNetError: 62020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.104756] LNet: Removed LNI 192.168.203.137@tcp [ 2002.892929] Key type .llcrypt unregistered [ 2002.894523] Key type ._llcrypt unregistered [ 2027.313352] Key type ._llcrypt registered [ 2027.315354] Key type .llcrypt registered [ 2027.406722] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2028.221333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.265597] alg: No test for adler32 (adler32-zlib) [ 2029.346308] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2029.610484] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2031.351856] Key type lgssc registered [ 2032.549757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.817631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.840964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.483278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.318383] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2052.578699] LustreError: 63432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.591444] LustreError: 63432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2055.339929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.691406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.109542] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2067.947904] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.361312] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.270149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.641553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.825093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2088.972518] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2098.019640] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.867957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2128.872495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.886730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.915392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.920410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.921664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.931757] Lustre: Skipped 1 previous similar message [ 2136.035992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2136.060284] Lustre: Skipped 1 previous similar message [ 2140.127108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2140.338930] Lustre: server umount lustre-OST0000 complete [ 2146.274402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.277145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.289047] Lustre: Skipped 1 previous similar message [ 2150.276156] Lustre: server umount lustre-MDT0000 complete [ 2151.392252] LustreError: 63432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.407025] LustreError: 63432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2153.674453] LustreError: 63412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441882 with bad export cookie 5925381097366739000 [ 2153.677682] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.685588] LustreError: 63412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.125203] Lustre: server umount lustre-MDT0001 complete [ 2166.137069] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2169.318894] Key type lgssc unregistered [ 2169.649036] LNet: 66990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.669786] LNetError: 66990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.686646] LNet: Removed LNI 192.168.203.137@tcp [ 2170.746211] Key type .llcrypt unregistered [ 2170.754845] Key type ._llcrypt unregistered [ 2196.986525] Key type ._llcrypt registered [ 2196.988204] Key type .llcrypt registered [ 2197.102848] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2198.282112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.325786] alg: No test for adler32 (adler32-zlib) [ 2199.358877] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2199.571532] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2201.215137] Key type lgssc registered [ 2202.450360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.856478] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:25:39 (1788441939) [ 2217.661347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2217.681284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.157614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.990968] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2229.731902] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.762140] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2232.654647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.073442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.858395] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2245.195859] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.939483] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.842633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.275345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.457297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2264.524448] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2272.339400] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2278.039380] Lustre: Failing over lustre-MDT0000 [ 2278.309300] Lustre: server umount lustre-MDT0000 complete [ 2279.401887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2279.411816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.443319] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.471466] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2279.908271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.923658] Lustre: Skipped 1 previous similar message [ 2284.575438] LustreError: 68687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.617881] LustreError: 68687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2287.634456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.783405] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.886278] Lustre: *** cfs_fail_loc=135, val=0*** [ 2287.972553] LustreError: 71776:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2287.978115] LustreError: 71776:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -2 [ 2287.990307] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2287.990307] [ 2288.006438] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2288.035709] LustreError: 71765:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2288.045527] LustreError: 71765:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2288.058709] LustreError: 71765:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2288.067787] LustreError: 71765:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2288.099285] Lustre: server umount lustre-MDT0000 complete [ 2288.104647] LustreError: 71765:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2289.668334] LustreError: 68685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.687357] LustreError: 68685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2294.647981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.788922] LustreError: 68687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.837959] LustreError: 68687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2294.884513] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.899968] LustreError: 68683:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2294.911093] LustreError: 72121:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000a26a4bb9) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2295.266497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.339408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2295.853397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2300.028241] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2300.404219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2300.425258] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2300.462088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2310.629330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.644840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2310.645305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.762443] Lustre: server umount lustre-OST0000 complete [ 2320.869075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.871096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.884615] Lustre: Skipped 1 previous similar message [ 2320.886069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.886075] Lustre: Skipped 1 previous similar message [ 2325.007424] Lustre: server umount lustre-MDT0000 complete [ 2325.990482] LustreError: 68691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.011482] LustreError: 68691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2328.702533] LustreError: 68670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442057 with bad export cookie 5127897165569854752 [ 2328.715565] LustreError: 68670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2328.718312] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2329.280878] Lustre: server umount lustre-MDT0001 complete [ 2341.208279] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2344.496344] Key type lgssc unregistered [ 2344.828710] LNet: 73555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.839904] LNetError: 73555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.864430] LNet: Removed LNI 192.168.203.137@tcp [ 2345.825637] Key type .llcrypt unregistered [ 2345.828584] Key type ._llcrypt unregistered [ 2369.969513] Key type ._llcrypt registered [ 2369.971671] Key type .llcrypt registered [ 2370.060858] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2370.978416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.072069] alg: No test for adler32 (adler32-zlib) [ 2372.118268] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2372.531746] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2374.351829] Key type lgssc registered [ 2375.694707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.054980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.070867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.517729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.293701] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2394.089806] LustreError: 74970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.110608] LustreError: 74970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2398.041119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.389846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.630629] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2409.880250] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.303431] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2421.461817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2421.718352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.713883] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2428.933312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2435.211980] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.829268] Lustre: Failing over lustre-MDT0000 [ 2442.112083] Lustre: server umount lustre-MDT0000 complete [ 2443.291921] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.264566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.273804] Lustre: Skipped 1 previous similar message [ 2444.293253] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.314829] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2448.398689] LustreError: 76402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.261854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.406381] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2452.553149] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2452.682751] LustreError: 78054:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2452.688170] LustreError: 78054:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -9 [ 2452.697302] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2452.697302] [ 2452.719703] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2452.741090] LustreError: 78042:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2452.751651] LustreError: 78042:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2452.761686] LustreError: 78042:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2452.770299] LustreError: 78042:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2452.804231] Lustre: server umount lustre-MDT0000 complete [ 2452.808054] LustreError: 78042:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2453.509871] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.539576] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2459.741309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.910654] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.924210] LustreError: 74961:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2460.200525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.269506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2463.753021] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2465.266787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2465.300651] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2465.360173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2465.696361] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2474.978256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2474.991407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.005295] Lustre: Skipped 1 previous similar message [ 2475.026604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.048895] Lustre: server umount lustre-OST0000 complete [ 2485.729356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2485.730667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2485.753887] Lustre: Skipped 2 previous similar messages [ 2485.771089] Lustre: Skipped 3 previous similar messages [ 2490.330733] Lustre: server umount lustre-MDT0000 complete [ 2490.853921] LustreError: 74969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.882673] LustreError: 74969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2493.947352] LustreError: 74949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442223 with bad export cookie 14604765086536398304 [ 2493.952982] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.955147] LustreError: 74949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2494.371110] Lustre: server umount lustre-MDT0001 complete [ 2505.718603] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2508.824279] Key type lgssc unregistered [ 2509.193079] LNet: 79834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.201555] LNetError: 79834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.218835] LNet: Removed LNI 192.168.203.137@tcp [ 2510.366749] Key type .llcrypt unregistered [ 2510.375180] Key type ._llcrypt unregistered [ 2524.267609] Key type ._llcrypt registered [ 2524.272566] Key type .llcrypt registered [ 2524.376337] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2524.757195] Key type .llcrypt unregistered [ 2524.760894] Key type ._llcrypt unregistered [ 2548.553658] Key type ._llcrypt registered [ 2548.559445] Key type .llcrypt registered [ 2548.700697] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2549.783696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.833642] alg: No test for adler32 (adler32-zlib) [ 2551.066770] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2551.305234] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2552.991268] Key type lgssc registered [ 2554.449841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.277429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.303732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.868670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.570387] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2573.296955] LustreError: 81731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.319515] LustreError: 81731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2576.428390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.866408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.501488] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2587.990063] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2594.113192] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2601.354606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.441176] LustreError: 83556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.820881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.110202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2608.687836] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2615.121238] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.460026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2623.465796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.499919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2623.504320] Lustre: Skipped 1 previous similar message [ 2628.583296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.596625] Lustre: Skipped 1 previous similar message [ 2629.413632] Lustre: server umount lustre-OST0000 complete [ 2633.696791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.722129] Lustre: Skipped 1 previous similar message [ 2633.732317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.311408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2638.328142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.352317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.668729] Lustre: server umount lustre-MDT0000 complete [ 2643.122354] LustreError: 81713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442372 with bad export cookie 16750610801092282550 [ 2643.130697] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.498745] Lustre: server umount lustre-MDT0001 complete [ 2654.828477] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2657.502161] Key type lgssc unregistered [ 2657.845973] LNet: 85266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2657.860509] LNetError: 85266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.896990] LNet: Removed LNI 192.168.203.137@tcp [ 2658.805378] Key type .llcrypt unregistered [ 2658.808267] Key type ._llcrypt unregistered [ 2682.000698] Key type ._llcrypt registered [ 2682.003971] Key type .llcrypt registered [ 2682.119880] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2683.291375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.332977] alg: No test for adler32 (adler32-zlib) [ 2684.498412] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2684.827105] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2686.551176] Key type lgssc registered [ 2687.663983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.966460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2695.986602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.395746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.009682] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2706.914105] LustreError: 86675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.928930] LustreError: 86675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2710.523082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.907253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.560684] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2723.066289] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2728.991979] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2736.566193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.099178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.232384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2744.448468] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2752.882561] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2762.721822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.735932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.752096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.793994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.797907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.816762] Lustre: Skipped 1 previous similar message [ 2768.346672] Lustre: server umount lustre-OST0000 complete [ 2772.452768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.463213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.486468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.032447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.033994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.083997] Lustre: Skipped 1 previous similar message [ 2777.920277] Lustre: server umount lustre-MDT0000 complete [ 2781.152140] LustreError: 87686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.171236] LustreError: 87686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2781.764030] LustreError: 87410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442510 with bad export cookie 14234803944580014119 [ 2781.777613] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2781.787698] LustreError: 87410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2782.122642] Lustre: server umount lustre-MDT0001 complete [ 2794.037177] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2797.344571] Key type lgssc unregistered [ 2797.666266] LNet: 90209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.675979] LNetError: 90209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.718327] LNet: Removed LNI 192.168.203.137@tcp [ 2798.874681] Key type .llcrypt unregistered [ 2798.878259] Key type ._llcrypt unregistered [ 2824.918346] Key type ._llcrypt registered [ 2824.921744] Key type .llcrypt registered [ 2825.045408] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2826.450392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2826.504074] alg: No test for adler32 (adler32-zlib) [ 2827.588857] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2827.911832] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2829.607554] Key type lgssc registered [ 2831.153246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2839.335569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2839.376353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.919728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.722902] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2851.297837] LustreError: 91629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.319837] LustreError: 91629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2853.819446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.133668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.362299] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2865.902567] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2871.442411] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2878.603305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.952028] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2879.107534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.857447] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 2886.640092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2894.134826] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.176419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.184315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.199451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.224390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.227214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.239759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.519734] Lustre: server umount lustre-OST0000 complete [ 2917.352250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.375803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.382311] Lustre: Skipped 2 previous similar messages [ 2922.464376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.483263] Lustre: Skipped 1 previous similar message [ 2923.195361] Lustre: server umount lustre-MDT0000 complete [ 2926.429047] LustreError: 91609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442655 with bad export cookie 16988367802248609977 [ 2926.448987] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.809446] Lustre: server umount lustre-MDT0001 complete [ 2937.465435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 2940.254245] Key type lgssc unregistered [ 2940.526556] LNet: 95183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2940.530920] LNetError: 95183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2941.608837] LNet: Removed LNI 192.168.203.137@tcp [ 2942.755396] Key type .llcrypt unregistered [ 2942.757670] Key type ._llcrypt unregistered [ 2970.711821] Key type ._llcrypt registered [ 2970.715549] Key type .llcrypt registered [ 2970.788193] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 2971.310886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2971.331719] alg: No test for adler32 (adler32-zlib) [ 2972.436774] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2972.778179] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 2974.431184] Key type lgssc registered [ 2975.752254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2984.140516] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2984.160926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.711693] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3004.831314] LustreError: 96565:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 3031.519253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.091266] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3043.024773] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3048.494288] Lustre: server umount lustre-OST0000 complete [ 3056.355946] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:39:44 (1788442784) [ 3062.305628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.715294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.154382] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3067.873732] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.908119] LustreError: 98205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3071.975549] LustreError: 98204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.190553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.551212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.118878] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3087.627498] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.033257] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3100.210975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.426524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.633155] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3108.679385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3114.252270] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3129.315474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.327696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.363596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.337680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.341675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.341844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.542178] Lustre: server umount lustre-OST0000 complete [ 3140.577104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.603541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.611223] Lustre: Skipped 2 previous similar messages [ 3145.695987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.820548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.832773] Lustre: Skipped 2 previous similar messages [ 3152.351196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3152.743319] Lustre: server umount lustre-MDT0000 complete [ 3155.936031] LustreError: 98932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.960403] LustreError: 98932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3157.523978] LustreError: 98184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442886 with bad export cookie 15051955350267328131 [ 3157.528979] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.547767] LustreError: 98184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3157.958881] Lustre: server umount lustre-MDT0001 complete [ 3167.178674] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3169.685633] Key type lgssc unregistered [ 3169.921499] LNet: 101896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.930546] LNetError: 101896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.945197] LNet: Removed LNI 192.168.203.137@tcp [ 3170.876749] Key type .llcrypt unregistered [ 3170.878757] Key type ._llcrypt unregistered [ 3217.048114] Key type ._llcrypt registered [ 3217.050550] Key type .llcrypt registered [ 3217.251967] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3218.370030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3218.406071] alg: No test for adler32 (adler32-zlib) [ 3219.501487] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3219.741885] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3221.471163] Key type lgssc registered [ 3222.454878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3230.802060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3230.835168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.506242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.001250] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3241.963261] LustreError: 104081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.995967] LustreError: 104081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3247.080025] LustreError: 104082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.160743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.432619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.994825] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3260.540455] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3266.275396] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.709412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.127423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.251192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3280.479291] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3288.739485] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3302.885596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.894856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.933273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3307.516256] Lustre: server umount lustre-OST0000 complete [ 3313.128763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.139494] Lustre: Skipped 1 previous similar message [ 3313.142757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.161612] Lustre: Skipped 2 previous similar messages [ 3317.851706] Lustre: server umount lustre-MDT0000 complete [ 3318.243329] LustreError: 104081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.259056] LustreError: 104081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3322.334974] LustreError: 104063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443051 with bad export cookie 10705757481019783088 [ 3322.346616] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.357592] LustreError: 104063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3323.069246] Lustre: server umount lustre-MDT0001 complete [ 3335.769447] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3338.647760] Key type lgssc unregistered [ 3338.884869] LNet: 107634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.898951] LNetError: 107634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3339.957938] LNet: Removed LNI 192.168.203.137@tcp [ 3340.870672] Key type .llcrypt unregistered [ 3340.874410] Key type ._llcrypt unregistered [ 3359.396253] Key type ._llcrypt registered [ 3359.399343] Key type .llcrypt registered [ 3359.522455] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3361.047446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.083446] alg: No test for adler32 (adler32-zlib) [ 3362.184517] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3362.493944] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3364.287172] Key type lgssc registered [ 3365.532420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3373.427544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3373.473470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.023928] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3394.147251] LustreError: 108877:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 3420.895189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.402855] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3434.162986] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3443.111952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.598724] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3443.615672] LustreError: 109909:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3443.626536] LustreError: 109909:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3443.638367] LustreError: 109909:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3443.828950] Lustre: server umount lustre-MDT0000 complete [ 3443.831146] LustreError: 109909:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3462.342861] Lustre: server umount lustre-OST0000 complete [ 3471.877708] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3481.620881] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3515.051301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.323328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3515.357487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3515.460804] Lustre: lustre-MDT0000: new disk, initializing [ 3515.549940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.561898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3520.414310] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3530.703810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.824775] Lustre: 112700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3530.903829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3530.907975] Lustre: Skipped 1 previous similar message [ 3530.983622] Lustre: lustre-MDT0001: new disk, initializing [ 3531.106271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.175816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3531.191660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3536.061347] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3544.634968] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3549.810338] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3555.955605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.219854] Lustre: lustre-OST0000: new disk, initializing [ 3556.229959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3556.241867] Lustre: 114670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3556.332608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.361705] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3565.657528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3565.697500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3565.885792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3571.117696] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3575.993649] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3576.219996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3580.725654] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3581.007501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3586.528552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3586.529152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.540450] LustreError: Skipped 1 previous similar message [ 3586.541723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3586.574613] Lustre: Skipped 1 previous similar message [ 3589.432086] Lustre: server umount lustre-OST0000 complete [ 3596.770027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.791132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.800973] Lustre: Skipped 2 previous similar messages [ 3599.484499] Lustre: server umount lustre-MDT0000 complete [ 3603.023470] LustreError: 112691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443332 with bad export cookie 6125657698933435622 [ 3603.030385] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3603.033829] LustreError: 112691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3603.287994] Lustre: server umount lustre-MDT0001 complete [ 3611.520241] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:48:59 (1788443339) [ 3614.527322] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3629.611423] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3639.905224] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3671.744488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.040328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3672.065951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3672.128481] Lustre: lustre-MDT0000: new disk, initializing [ 3672.205139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.250120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3676.556426] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3686.785501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.918687] Lustre: 119571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3686.965934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3686.969756] Lustre: Skipped 1 previous similar message [ 3687.033855] Lustre: lustre-MDT0001: new disk, initializing [ 3687.115154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3687.152715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3687.162944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3691.774255] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3700.015677] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3705.038055] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3710.842544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.104702] Lustre: lustre-OST0000: new disk, initializing [ 3711.108082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3711.115513] Lustre: 121536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3711.196664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.371936] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3720.289452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3720.306118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3720.398641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3725.357477] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3729.152993] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3729.344641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3732.746658] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3732.902252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3736.035373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.038028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.040345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.050382] LustreError: Skipped 1 previous similar message [ 3736.059653] Lustre: Skipped 2 previous similar messages [ 3740.694598] Lustre: server umount lustre-OST0000 complete [ 3746.273138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.280098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.285424] Lustre: Skipped 2 previous similar messages [ 3750.862421] Lustre: server umount lustre-MDT0000 complete [ 3751.393592] LustreError: 122595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.411695] LustreError: 122595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3754.529496] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443483 with bad export cookie 6125657698933436504 [ 3754.531907] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.537836] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3754.958377] Lustre: server umount lustre-MDT0001 complete [ 3762.997606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.283406] LustreError: 123751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.359994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.677870] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3768.804325] LustreError: 123752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.900439] LustreError: 123751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.081800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.720200] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3786.163318] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3791.254714] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3797.347088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3803.549436] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3813.525973] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.032614] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3826.659439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.667163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.679658] Lustre: Skipped 1 previous similar message [ 3826.690615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.735432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3831.752578] Lustre: server umount lustre-OST0000 complete [ 3836.897042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.900440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.906808] Lustre: Skipped 1 previous similar message [ 3836.910190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.913656] Lustre: Skipped 2 previous similar messages [ 3841.895212] Lustre: server umount lustre-MDT0000 complete [ 3845.096554] LustreError: 125343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.108418] LustreError: 125343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3845.746536] LustreError: 123733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443574 with bad export cookie 6125657698933437386 [ 3845.750534] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3845.756890] LustreError: 123733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3846.122099] Lustre: server umount lustre-MDT0001 complete [ 3863.819858] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 3866.813511] Key type lgssc unregistered [ 3867.069832] LNet: 127399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.076346] LNetError: 127399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.087131] LNet: Removed LNI 192.168.203.137@tcp [ 3867.755749] Key type .llcrypt unregistered [ 3867.760645] Key type ._llcrypt unregistered [ 3883.252549] Key type ._llcrypt registered [ 3883.257684] Key type .llcrypt registered [ 3883.374599] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 3898.724552] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 3900.063991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3900.147480] alg: No test for adler32 (adler32-zlib) [ 3901.415710] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3901.698718] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 3903.375160] Key type lgssc registered [ 3904.792858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3940.994466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3941.049855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.350788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3942.414675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3942.501791] Lustre: lustre-MDT0000: new disk, initializing [ 3942.600636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.629754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3947.784316] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3959.429955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.534209] Lustre: 130545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3959.627125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3959.633199] Lustre: Skipped 1 previous similar message [ 3959.757599] Lustre: lustre-MDT0001: new disk, initializing [ 3959.871120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3959.942352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3959.965813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3966.127876] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 3977.404783] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3983.742550] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3991.768623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3992.162564] Lustre: lustre-OST0000: new disk, initializing [ 3992.175311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3992.192667] Lustre: 132514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3992.345380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.869355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3998.898419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3999.149735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3999.952755] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4009.239685] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4014.198608] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4014.385995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4018.691369] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4018.935829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4024.801705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.808515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.817057] LustreError: Skipped 1 previous similar message [ 4024.817945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.846328] Lustre: Skipped 1 previous similar message [ 4027.303412] Lustre: server umount lustre-OST0000 complete [ 4035.042094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.050057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4035.078297] Lustre: Skipped 1 previous similar message [ 4035.101816] Lustre: Skipped 2 previous similar messages [ 4038.395367] Lustre: server umount lustre-MDT0000 complete [ 4043.729610] LustreError: 131340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443772 with bad export cookie 9765508366922667597 [ 4043.745331] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.758284] LustreError: 131340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4044.216137] Lustre: server umount lustre-MDT0001 complete [ 4054.117350] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:56:22 (1788443782) [ 4060.129573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.762918] LustreError: 134928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.781765] LustreError: 134928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4060.874246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.047517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4066.277040] LustreError: 134929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.375793] LustreError: 134928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.386420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.765530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4078.492188] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4085.839920] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.683570] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4093.630154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.634118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.641598] Lustre: Skipped 1 previous similar message [ 4098.785976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.793071] Lustre: Skipped 2 previous similar messages [ 4100.236407] Lustre: server umount lustre-MDT0000 complete [ 4104.165726] LustreError: 134928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.183116] LustreError: 134928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4104.731597] LustreError: 134909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443833 with bad export cookie 9765508366922668486 [ 4104.748661] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4104.757157] LustreError: 134909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4105.345628] Lustre: server umount lustre-MDT0001 complete [ 4114.935458] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:57:23 (1788443843) [ 4121.932252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4137.567635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4142.687647] LustreError: 137344:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 4168.416783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.850727] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4182.983177] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.471317] Lustre: server umount lustre-OST0000 complete [ 4193.805654] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:58:41 (1788443921) [ 4200.442093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.912175] LustreError: 138792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.029037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.505122] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4211.174124] LustreError: 138792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.195177] LustreError: 138792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4214.406553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.475214] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4229.087220] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4235.386200] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4242.166333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4242.649824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.659439] Lustre: Skipped 1 previous similar message [ 4249.557577] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4256.827994] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4265.956880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4265.987691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.010583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4271.565867] Lustre: server umount lustre-MDT0000 complete [ 4274.674325] LustreError: 140449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.715907] LustreError: 140449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4275.747587] LustreError: 138771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444004 with bad export cookie 9765508366922669151 [ 4275.754940] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4275.769674] LustreError: 138771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4276.044899] Lustre: server umount lustre-MDT0001 complete [ 4289.661761] Lustre: server umount lustre-OST0000 complete [ 4302.316272] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:00:29 (1788444029) [ 4311.599829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.288635] LustreError: 142446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.417948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.914080] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4325.733230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.782860] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4338.790490] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4345.075569] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.549755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.126670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4358.927987] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4367.620723] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.992746] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4373.333241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4379.328666] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4379.516637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4382.689909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4382.712977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.759280] Lustre: Skipped 2 previous similar messages [ 4382.772637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.775820] Lustre: Skipped 3 previous similar messages [ 4386.803422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.293463] Lustre: server umount lustre-OST0000 complete [ 4392.928837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4392.937328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.946844] Lustre: Skipped 1 previous similar message [ 4402.145676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4402.148927] Lustre: Skipped 6 previous similar messages [ 4407.263435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4407.661517] Lustre: server umount lustre-MDT0000 complete [ 4412.389337] LustreError: 142446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.415607] LustreError: 142446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4412.498287] LustreError: 142428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444141 with bad export cookie 9765508366922670222 [ 4412.518434] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.843625] Lustre: server umount lustre-MDT0001 complete [ 4421.992315] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:02:30 (1788444150) [ 4429.131524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.639617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4449.759730] LustreError: 146616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 4475.554516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.564558] Lustre: Skipped 2 previous similar messages [ 4482.756602] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4490.559658] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4497.299497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.223048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4502.236575] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4511.509554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.618498] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4524.098885] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.147848] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4533.963976] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4534.212936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4538.532128] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4538.834262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4542.947093] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.948133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4542.950317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4542.950324] Lustre: Skipped 2 previous similar messages [ 4542.975415] Lustre: Skipped 2 previous similar messages [ 4555.231136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4555.438582] Lustre: server umount lustre-MDT0000 complete [ 4558.309421] LustreError: 147654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.367950] LustreError: 147654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4559.447775] LustreError: 146622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444288 with bad export cookie 9765508366922671244 [ 4559.463811] LustreError: 146622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4559.486864] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.798805] Lustre: server umount lustre-MDT0001 complete [ 4574.167125] Lustre: server umount lustre-OST0000 complete [ 4583.531768] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:05:11 (1788444311) [ 4591.378664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.071630] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4612.191475] LustreError: 150814:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 4645.001935] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4654.200712] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4662.292685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.132949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4668.655984] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4677.515814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.969754] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4690.962927] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.559761] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.295388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4704.526825] Lustre: lustre-OST0001: new disk, initializing [ 4704.531313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4704.542633] Lustre: 153690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4711.517596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4714.640991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4714.648697] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4714.717063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4721.535710] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.516153] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4727.805325] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4733.035645] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4733.337592] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4735.457858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.470337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4735.480982] Lustre: Skipped 2 previous similar messages [ 4735.503698] Lustre: Skipped 9 previous similar messages [ 4741.459651] Lustre: server umount lustre-OST0000 complete [ 4745.699405] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4745.712428] LustreError: Skipped 1 previous similar message [ 4751.839147] Lustre: server umount lustre-OST0001 complete [ 4761.482872] Lustre: server umount lustre-MDT0000 complete [ 4766.011766] LustreError: 150821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444495 with bad export cookie 9765508366922672196 [ 4766.014531] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4766.020396] LustreError: 150821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4766.607330] Lustre: server umount lustre-MDT0001 complete [ 4791.054691] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:08:39 (1788444519) [ 4792.977871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4794.482815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4796.369869] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:08:44 (1788444524) [ 4803.449333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.787274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4803.813895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4804.011031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.017497] Lustre: Skipped 6 previous similar messages [ 4808.860571] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4817.820930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.970863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4818.016288] Lustre: 157250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4822.482927] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4828.997414] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4835.040488] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4842.280260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4842.567226] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4848.827338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4849.831935] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4859.823459] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.723069] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4864.928367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4870.036657] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4870.333307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4874.723846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4874.730955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.740690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4874.740701] Lustre: Skipped 10 previous similar messages [ 4874.769441] Lustre: Skipped 5 previous similar messages [ 4878.799451] Lustre: server umount lustre-OST0000 complete [ 4901.752363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4909.016904] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 4914.418204] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.436544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.439898] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4914.480714] Lustre: Skipped 1 previous similar message [ 4917.710449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4923.226957] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4923.514457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4928.727820] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4929.023953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4934.900763] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90cf86fe1000.ost_server_uuid 50 [ 4936.499452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90cf86fe1000.ost_server_uuid in FULL state after 0 sec [ 4936.690068] LustreError: 157255:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.37@tcp) returned error from blocking AST (req@000000003ecd173a x1875319068040960 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000445a9ec6/0x87860e1ead87f19e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.37@tcp remote: 0xeb7a95743791d30 expref: 5 pid: 157255 timeout: 5036 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4936.733291] LustreError: lustre-MDT0000: A client on nid 192.168.203.37@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4936.744098] LustreError: 157246:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.37@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000445a9ec6/0x87860e1ead87f19e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.37@tcp remote: 0xeb7a95743791d30 expref: 6 pid: 157255 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4942.833426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4942.846341] LustreError: Skipped 1 previous similar message [ 4948.335231] Lustre: server umount lustre-OST0000 complete [ 4958.783879] Lustre: server umount lustre-MDT0000 complete [ 4960.737378] LustreError: 157257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.763386] LustreError: 157257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4962.574051] LustreError: 157243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444691 with bad export cookie 9765508366922673379 [ 4962.581279] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4962.586824] LustreError: 157243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4968.888324] Lustre: server umount lustre-MDT0001 complete [ 4979.418965] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 4982.321376] Key type lgssc unregistered [ 4982.580872] LNet: 163363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4982.596659] LNetError: 163363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4983.657180] LNet: Removed LNI 192.168.203.137@tcp [ 4984.606928] Key type .llcrypt unregistered [ 4984.612968] Key type ._llcrypt unregistered [ 5009.070731] Key type ._llcrypt registered [ 5009.077219] Key type .llcrypt registered [ 5009.295074] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5011.000497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5011.078189] alg: No test for adler32 (adler32-zlib) [ 5012.292803] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5012.665837] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5014.311453] Key type lgssc registered [ 5015.967159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5026.283474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5026.299537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.699355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.450788] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5037.032492] LustreError: 164774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.058510] LustreError: 164774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5042.130411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.145575] LustreError: 164773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.543464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5047.151165] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5055.110181] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5061.205074] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5068.602846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.077654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5072.205725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5076.230222] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5084.734291] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5094.344057] Lustre: Failing over lustre-MDT0000 [ 5094.799758] Lustre: server umount lustre-MDT0000 complete [ 5097.955861] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.964624] LustreError: 164774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.973698] Lustre: Skipped 2 previous similar messages [ 5098.003093] LustreError: 164774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5103.076781] LustreError: 164769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.110672] LustreError: 164769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5106.970458] LustreError: 165511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.37@tcp arrived at 1788444836 with bad export cookie 11362569238353865894 [ 5107.001066] LustreError: 165511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5108.197561] LustreError: 167273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.214354] LustreError: 167273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5113.722838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.868726] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.172666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.244409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5118.931991] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5119.470913] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5119.475879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5121.303693] Lustre: lustre-MDT0000: Denying connection for new client 597ad8aa-ce1c-4395-afcf-e56f0294a8b8 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5133.793792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5133.808335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.830959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5134.816374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5134.826175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.940345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5139.952470] Lustre: Skipped 2 previous similar messages [ 5144.031253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5144.344282] Lustre: server umount lustre-OST0000 complete [ 5148.047865] LustreError: 168505:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5148.062749] Lustre: 167853:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5148.075448] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5148.096221] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5148.103636] LustreError: 167853:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000092d92061 x1875320232786688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5148.124264] LustreError: 167853:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5148.138146] LustreError: 167853:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5148.148477] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5148.392907] Lustre: server umount lustre-MDT0000 complete [ 5150.180678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.196126] LustreError: 166271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.211531] LustreError: 166271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5152.104715] LustreError: 164755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444881 with bad export cookie 11362569238353865999 [ 5152.118464] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5152.126129] LustreError: 164755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5154.659424] Lustre: server umount lustre-MDT0001 complete [ 5168.794365] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5171.985228] Key type lgssc unregistered [ 5172.315086] LNet: 169259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5172.322637] LNetError: 169259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5172.352291] LNet: Removed LNI 192.168.203.137@tcp [ 5173.561824] Key type .llcrypt unregistered [ 5173.563358] Key type ._llcrypt unregistered [ 5200.876660] Key type ._llcrypt registered [ 5200.880190] Key type .llcrypt registered [ 5201.055601] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5201.860206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5201.882345] alg: No test for adler32 (adler32-zlib) [ 5203.097173] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5203.385113] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5205.047162] Key type lgssc registered [ 5206.508866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5216.497720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5216.540976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.531810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.781924] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5228.011546] LustreError: 170667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.024521] LustreError: 170667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5232.424994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.879911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.852727] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5246.954025] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5253.980705] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5261.364501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.667973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.730517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5266.864142] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5274.513851] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5282.273896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.275693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.299963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.315803] Lustre: Skipped 1 previous similar message [ 5286.759685] Lustre: server umount lustre-OST0000 complete [ 5292.512671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.536174] Lustre: Skipped 1 previous similar message [ 5292.541547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5292.548681] Lustre: Skipped 1 previous similar message [ 5296.799714] Lustre: server umount lustre-MDT0000 complete [ 5297.633315] LustreError: 171396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.662269] LustreError: 171396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5300.302201] LustreError: 170648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445029 with bad export cookie 6865067197553627555 [ 5300.318052] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5300.328643] LustreError: 170648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5300.690284] Lustre: server umount lustre-MDT0001 complete [ 5316.058179] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5321.180099] Key type lgssc unregistered [ 5321.778338] LNet: 174201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5321.793910] LNetError: 174201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.833232] LNet: Removed LNI 192.168.203.137@tcp [ 5323.183620] Key type .llcrypt unregistered [ 5323.187751] Key type ._llcrypt unregistered [ 5356.927237] Key type ._llcrypt registered [ 5356.930837] Key type .llcrypt registered [ 5357.108620] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5358.465864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5358.533831] alg: No test for adler32 (adler32-zlib) [ 5359.799866] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5360.141593] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5361.871265] Key type lgssc registered [ 5363.076123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5369.845266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5369.868704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.418035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5376.679905] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5381.601733] LustreError: 175902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.630131] LustreError: 175902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5385.961988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.413512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5392.202411] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5400.521237] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5405.016296] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5411.515768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.797259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.936569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5418.718312] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5426.899344] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5439.976178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5439.985418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.996690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.517359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.520248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.525211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.550503] Lustre: Skipped 1 previous similar message [ 5446.624842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.635153] Lustre: Skipped 1 previous similar message [ 5451.764443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.776599] Lustre: Skipped 1 previous similar message [ 5452.255362] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5452.399150] Lustre: server umount lustre-OST0000 complete [ 5457.889609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5457.896597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.904485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5461.986406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.359309] Lustre: server umount lustre-MDT0000 complete [ 5467.105040] LustreError: 175902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.139577] LustreError: 175902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5467.483230] LustreError: 176637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445196 with bad export cookie 3438684290507361721 [ 5467.497265] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5467.508346] LustreError: 176637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5467.952577] Lustre: server umount lustre-MDT0001 complete [ 5479.094259] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5482.089246] Key type lgssc unregistered [ 5482.418892] LNet: 179437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5482.428141] LNetError: 179437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5482.445768] LNet: Removed LNI 192.168.203.137@tcp [ 5483.360993] Key type .llcrypt unregistered [ 5483.362727] Key type ._llcrypt unregistered [ 5506.107628] Key type ._llcrypt registered [ 5506.109580] Key type .llcrypt registered [ 5506.221255] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5507.312248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5507.391693] alg: No test for adler32 (adler32-zlib) [ 5508.463335] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5508.729392] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5510.383161] Key type lgssc registered [ 5511.479772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5522.644619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5522.686541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.098695] Lustre: *** cfs_fail_loc=135, val=0*** [ 5524.230160] LustreError: 180958:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5524.235617] LustreError: 180958:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.137@tcp: cfg command failed: rc = -2 [ 5524.251290] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5524.251290] [ 5524.260684] LustreError: MGC192.168.203.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5524.275652] LustreError: 180919:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5524.286211] LustreError: 180919:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5524.295613] LustreError: 180919:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5524.303373] LustreError: 180919:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5524.388164] Lustre: server umount lustre-MDT0000 complete [ 5524.391432] LustreError: 180919:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5530.689705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.132324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.461947] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5541.352762] LustreError: 181330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5541.370943] LustreError: 181330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5544.140271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.397571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5549.359638] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5557.034967] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5564.575625] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5566.949084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.958455] Lustre: Skipped 1 previous similar message [ 5566.965764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.986348] Lustre: Skipped 1 previous similar message [ 5569.058869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.071643] Lustre: Skipped 1 previous similar message [ 5574.116329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.131404] Lustre: Skipped 1 previous similar message [ 5579.234789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5579.241035] Lustre: Skipped 1 previous similar message [ 5581.279292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5581.553557] Lustre: server umount lustre-MDT0000 complete [ 5584.353240] LustreError: 181331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.365963] LustreError: 181331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5584.947679] LustreError: 182065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445314 with bad export cookie 16116793334108041596 [ 5584.954089] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.958261] LustreError: 182065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5585.399265] Lustre: server umount lustre-MDT0001 complete [ 5596.095228] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5598.970673] Key type lgssc unregistered [ 5599.312456] LNet: 183649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.330988] LNetError: 183649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5600.360469] LNet: Removed LNI 192.168.203.137@tcp [ 5601.355780] Key type .llcrypt unregistered [ 5601.357796] Key type ._llcrypt unregistered [ 5623.933374] Key type ._llcrypt registered [ 5623.937944] Key type .llcrypt registered [ 5624.093722] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5624.447566] Key type .llcrypt unregistered [ 5624.450382] Key type ._llcrypt unregistered [ 5641.202851] Key type ._llcrypt registered [ 5641.206155] Key type .llcrypt registered [ 5641.323380] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5642.602799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5642.652875] alg: No test for adler32 (adler32-zlib) [ 5643.814545] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5644.035605] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5645.704250] Key type lgssc registered [ 5646.907777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5654.543866] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5654.573576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.175661] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5675.295266] LustreError: 185775:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.137@tcp: failed processing log, type 4: rc = -110 [ 5701.984939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.385177] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5714.864388] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5721.595628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.124923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.185681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5726.701826] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5727.205546] LustreError: 185800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.227429] LustreError: 185800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5732.321710] LustreError: 185801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.002185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.377774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.607568] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5748.304834] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5756.348363] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5760.306565] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5771.238824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.243780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.254208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5772.259802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5772.260705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.385905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.391530] Lustre: Skipped 1 previous similar message [ 5782.508713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5782.522859] Lustre: Skipped 1 previous similar message [ 5783.007453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5783.108878] Lustre: server umount lustre-OST0000 complete [ 5787.629152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.630962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.659783] Lustre: Skipped 1 previous similar message [ 5787.682603] Lustre: Skipped 1 previous similar message [ 5793.476349] Lustre: server umount lustre-MDT0000 complete [ 5796.989938] LustreError: 185782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445526 with bad export cookie 5209056658049306795 [ 5796.994778] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.998541] LustreError: 185782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5797.575271] Lustre: server umount lustre-MDT0001 complete [ 5808.680729] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 5811.415741] Key type lgssc unregistered [ 5811.731874] LNet: 189689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5811.750062] LNetError: 189689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5811.767928] LNet: Removed LNI 192.168.203.137@tcp [ 5812.651555] Key type .llcrypt unregistered [ 5812.659616] Key type ._llcrypt unregistered [ 5835.812946] Key type ._llcrypt registered [ 5835.815038] Key type .llcrypt registered [ 5835.959768] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 5836.827325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5836.928584] alg: No test for adler32 (adler32-zlib) [ 5838.055930] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5838.272433] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 5839.935198] Key type lgssc registered [ 5841.049146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5850.106906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5850.178568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.845797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.198506] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5861.365311] LustreError: 191101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.412212] LustreError: 191101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5866.487942] LustreError: 191100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.021973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.699732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.491130] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5886.412539] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.548086] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5899.400974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5899.797862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.938271] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5913.694323] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5922.314997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5929.263630] Lustre: Failing over lustre-MDT0000 [ 5929.712155] Lustre: server umount lustre-MDT0000 complete [ 5930.569566] LustreError: 191096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.514185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.529017] Lustre: Skipped 2 previous similar messages [ 5935.626725] LustreError: 191094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.651596] LustreError: 191094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5940.742984] LustreError: 191096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.782451] LustreError: 191096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5947.880142] Lustre: 190412:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445661/real 1788445661] req@00000000f84cde15 x1875321098893568/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5947.925568] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.174977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.988858] LustreError: 191094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.006080] LustreError: 191094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5958.117598] LustreError: 190409:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce758be2 x1875321098899072/t0(0) o250->MGC192.168.203.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5958.579920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.640407] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5962.250114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5963.759928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5963.790953] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5963.860538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5963.982389] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 5968.583716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5972.412249] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5979.618271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.625054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.648633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5981.972610] Lustre: server umount lustre-OST0000 complete [ 5989.346754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.348881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.364951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.370844] Lustre: Skipped 1 previous similar message [ 5993.314724] Lustre: server umount lustre-MDT0000 complete [ 5994.477300] LustreError: 191100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5994.486536] LustreError: 191100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5997.391748] LustreError: 191081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445726 with bad export cookie 5950425963165357369 [ 5997.400245] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.409923] LustreError: 191081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6007.981598] Lustre: server umount lustre-MDT0001 complete [ 6018.614626] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6021.694451] Key type lgssc unregistered [ 6021.998355] LNet: 195912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.006425] LNetError: 195912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.030515] LNet: Removed LNI 192.168.203.137@tcp [ 6022.965748] Key type .llcrypt unregistered [ 6022.968621] Key type ._llcrypt unregistered [ 6047.951633] Key type ._llcrypt registered [ 6047.955314] Key type .llcrypt registered [ 6048.035932] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6048.689495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.714859] alg: No test for adler32 (adler32-zlib) [ 6049.787538] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6049.987730] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6051.615188] Key type lgssc registered [ 6052.708406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.895978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6060.934172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.446935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.356547] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6072.823085] LustreError: 197337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.851193] LustreError: 197337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6076.667662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.118656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6081.852790] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6089.376312] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6094.814677] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.828365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6102.007443] LustreError: 199158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.305987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.457781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6108.891488] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6116.788397] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6119.760320] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6128.440920] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6144.054530] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6153.697840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.699035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.707068] Lustre: Skipped 1 previous similar message [ 6157.822406] Lustre: server umount lustre-OST0000 complete [ 6163.941901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.945215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6163.960105] Lustre: Skipped 1 previous similar message [ 6163.978450] Lustre: Skipped 2 previous similar messages [ 6167.286749] Lustre: server umount lustre-MDT0000 complete [ 6169.058170] LustreError: 198349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.081974] LustreError: 198349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6170.539857] LustreError: 197320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445899 with bad export cookie 10257313539276580665 [ 6170.541565] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6170.548281] LustreError: 197320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6171.035866] Lustre: server umount lustre-MDT0001 complete [ 6181.973582] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6187.389565] Key type lgssc unregistered [ 6187.933696] LNet: 201210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.952227] LNetError: 201210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6187.970903] LNet: Removed LNI 192.168.203.137@tcp [ 6189.104185] Key type .llcrypt unregistered [ 6189.107727] Key type ._llcrypt unregistered [ 6209.787311] Key type ._llcrypt registered [ 6209.794151] Key type .llcrypt registered [ 6209.917542] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6211.146079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6211.251213] alg: No test for adler32 (adler32-zlib) [ 6212.339603] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6212.622887] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6214.360163] Key type lgssc registered [ 6215.469872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6223.238314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6223.259887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.680911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.177728] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6234.097811] LustreError: 202478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.121064] LustreError: 202478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6237.324915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.681777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.193054] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6249.384510] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6255.256980] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6261.934398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6262.206280] LustreError: 204298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.386487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.056475] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6268.914408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6276.593829] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6283.968489] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6293.664903] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6303.603499] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6309.099474] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6319.076364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.098459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.110872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.110641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.131878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.136754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.229226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6325.237618] Lustre: Skipped 2 previous similar messages [ 6330.341314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.350857] Lustre: Skipped 1 previous similar message [ 6332.386315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6332.498581] Lustre: server umount lustre-OST0000 complete [ 6340.068060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6340.080946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.129132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.934945] Lustre: server umount lustre-MDT0000 complete [ 6345.704530] LustreError: 202474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.725729] LustreError: 202474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6347.032351] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446076 with bad export cookie 17078275979007883377 [ 6347.034198] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.055256] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6347.488271] Lustre: server umount lustre-MDT0001 complete [ 6360.554551] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6364.224864] Key type lgssc unregistered [ 6364.483859] LNet: 207356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6364.500807] LNetError: 207356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6364.519502] LNet: Removed LNI 192.168.203.137@tcp [ 6365.644360] Key type .llcrypt unregistered [ 6365.646850] Key type ._llcrypt unregistered [ 6392.059436] Key type ._llcrypt registered [ 6392.062694] Key type .llcrypt registered [ 6392.173401] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6393.229963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6393.449367] alg: No test for adler32 (adler32-zlib) [ 6394.576383] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6394.870557] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6396.551174] Key type lgssc registered [ 6397.787425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6405.884938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6405.896567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.299395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.083088] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6416.876756] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.907979] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6420.712596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.083378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.362722] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6434.331852] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6440.809482] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6448.496482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.018009] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.089822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6456.252916] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6464.351949] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.892882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.897555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6481.919227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6481.944882] Lustre: Skipped 1 previous similar message [ 6486.923589] Lustre: server umount lustre-OST0000 complete [ 6492.130901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.138601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6492.159438] Lustre: Skipped 2 previous similar messages [ 6492.177972] Lustre: Skipped 1 previous similar message [ 6497.257898] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.283590] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6497.305987] Lustre: server umount lustre-MDT0000 complete [ 6501.208864] LustreError: 208746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446230 with bad export cookie 12478241238839035391 [ 6501.218230] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.223610] LustreError: 208746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6501.723931] Lustre: server umount lustre-MDT0001 complete [ 6513.379413] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6516.957574] Key type lgssc unregistered [ 6517.236506] LNet: 212449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6517.258703] LNetError: 212449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6517.296458] LNet: Removed LNI 192.168.203.137@tcp [ 6518.375603] Key type .llcrypt unregistered [ 6518.378502] Key type ._llcrypt unregistered [ 6537.690412] Key type ._llcrypt registered [ 6537.694184] Key type .llcrypt registered [ 6537.795455] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6539.200413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6539.233893] alg: No test for adler32 (adler32-zlib) [ 6540.365912] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6540.706697] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6542.432819] Key type lgssc registered [ 6543.757154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6551.407372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6551.430869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.793061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.332959] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6562.276879] LustreError: 213708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.291714] LustreError: 213708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6565.185680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.601111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6570.017551] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6577.321853] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6582.593623] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6588.596104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.871629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.478956] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6597.110192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6602.477405] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.521702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6615.530133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.543406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6617.573038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6617.576101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6617.580338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.634355] Lustre: Skipped 1 previous similar message [ 6620.408368] Lustre: server umount lustre-OST0000 complete [ 6627.296851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6627.305448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.320368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.929192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.937277] Lustre: Skipped 2 previous similar messages [ 6638.047610] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6638.049946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6638.082803] Lustre: Skipped 1 previous similar message [ 6638.251193] Lustre: server umount lustre-MDT0000 complete [ 6641.910780] LustreError: 213689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446371 with bad export cookie 1701154851451250969 [ 6641.914384] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.927808] LustreError: 213689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6642.252615] Lustre: server umount lustre-MDT0001 complete [ 6654.539126] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6658.567925] Key type lgssc unregistered [ 6659.011772] LNet: 217393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6659.016314] LNetError: 217393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6659.057199] LNet: Removed LNI 192.168.203.137@tcp [ 6660.217150] Key type .llcrypt unregistered [ 6660.222537] Key type ._llcrypt unregistered [ 6679.118343] Key type ._llcrypt registered [ 6679.122743] Key type .llcrypt registered [ 6679.247369] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6680.356022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6680.410585] alg: No test for adler32 (adler32-zlib) [ 6681.427141] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6681.610152] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6683.232461] Key type lgssc registered [ 6684.413828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6691.632779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6691.648477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.047373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.463789] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6703.597655] LustreError: 218650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6703.619293] LustreError: 218650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6705.868582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.269175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.452664] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6717.861350] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6723.303430] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6729.505044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6729.873743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.828346] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6738.106143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6742.329658] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6750.371311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6750.442218] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6750.646276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6756.595840] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6763.004176] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.549842] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6776.453573] Lustre: Permanently deactivating lustre-OST0001 [ 6776.482727] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6783.056818] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6790.095432] Lustre: Permanently reactivating lustre-OST0001 [ 6790.110735] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6790.126770] Lustre: Skipped 2 previous similar messages [ 6791.326718] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.342391] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6791.349722] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6791.362415] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6791.420230] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6801.889687] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6801.895679] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.895918] LustreError: Skipped 1 previous similar message [ 6801.896560] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6801.913282] Lustre: Skipped 2 previous similar messages [ 6803.552280] Lustre: server umount lustre-OST0001 complete [ 6805.475638] LustreError: 220474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.501504] LustreError: 220474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6808.032327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6808.038422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.058269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6808.061938] Lustre: Skipped 1 previous similar message [ 6810.081407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6810.091933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.114467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6810.618180] LustreError: 220565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.628991] LustreError: 220565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6813.369887] Lustre: server umount lustre-OST0000 complete [ 6818.784344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6818.798463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.821927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.833624] Lustre: Skipped 1 previous similar message [ 6823.170639] Lustre: server umount lustre-MDT0000 complete [ 6825.952830] LustreError: 219379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.966124] LustreError: 219379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6826.627952] LustreError: 219386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446555 with bad export cookie 10054772687759595445 [ 6826.631830] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.639032] LustreError: 219386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6826.982506] Lustre: server umount lustre-MDT0001 complete [ 6838.469711] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6841.338565] Key type lgssc unregistered [ 6841.643548] LNet: 223697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6841.664382] LNetError: 223697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6841.693768] LNet: Removed LNI 192.168.203.137@tcp [ 6842.637940] Key type .llcrypt unregistered [ 6842.640647] Key type ._llcrypt unregistered [ 6879.316962] Key type ._llcrypt registered [ 6879.321546] Key type .llcrypt registered [ 6879.444614] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 6880.581286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.784941] alg: No test for adler32 (adler32-zlib) [ 6881.968426] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6882.263981] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 6883.943359] Key type lgssc registered [ 6885.278157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6892.603563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6892.618239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.873584] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6893.901699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6894.181173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6898.490696] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6906.173653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.286680] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6906.308122] Lustre: 225897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6906.622650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6910.776424] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6917.481782] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.136062] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6927.708349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6927.848144] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6928.046150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.029368] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 6938.224320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6939.610284] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.074324] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6950.634616] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6961.009996] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6967.882483] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6983.743752] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6983.753435] Lustre: Skipped 1 previous similar message [ 7004.584398] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7004.589847] Lustre: Skipped 2 previous similar messages [ 7042.228945] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7042.235305] Lustre: Skipped 4 previous similar messages [ 7112.161031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.175262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.189692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.673187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.694909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.712351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7114.847714] Lustre: server umount lustre-OST0000 complete [ 7121.892694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7121.910971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7121.933340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7125.389638] Lustre: server umount lustre-MDT0000 complete [ 7128.839294] LustreError: 228468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446857 with bad export cookie 14153112841560603009 [ 7128.847417] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7128.852871] LustreError: 228468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7129.264458] Lustre: server umount lustre-MDT0001 complete [ 7139.552059] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7142.437836] Key type lgssc unregistered [ 7142.772131] LNet: 230688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7142.784514] LNetError: 230688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7142.800359] LNet: Removed LNI 192.168.203.137@tcp [ 7143.630285] Key type .llcrypt unregistered [ 7143.632394] Key type ._llcrypt unregistered [ 7167.697727] Key type ._llcrypt registered [ 7167.716250] Key type .llcrypt registered [ 7167.864715] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7169.003553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7169.097194] alg: No test for adler32 (adler32-zlib) [ 7170.220523] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7170.685730] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7172.495423] Key type lgssc registered [ 7173.770639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7181.396231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7181.412551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.998282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.501450] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7193.576762] LustreError: 232099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.591747] LustreError: 232099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7195.566844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.855273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7200.157261] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7206.637981] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.437838] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7216.832798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.137252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.798070] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug -1 all [ 7228.404689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7229.546827] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.300845] Lustre: server umount lustre-OST0000 complete [ 7248.879589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.886832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7248.889402] Lustre: Skipped 1 previous similar message [ 7248.909081] Lustre: Skipped 1 previous similar message [ 7253.985601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.001527] Lustre: Skipped 1 previous similar message [ 7258.200397] Lustre: server umount lustre-MDT0000 complete [ 7259.104105] LustreError: 232099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.127535] LustreError: 232099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7261.024831] Lustre: 231410:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446974/real 1788446974] req@00000000052a1615 x1875322495600384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788446990 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7261.057276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.070565] LustreError: 232838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7261.090293] LustreError: 232838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7261.771614] LustreError: 232080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446990 with bad export cookie 16878683587832088340 [ 7261.777450] LustreError: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.778200] LustreError: 232080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7262.104518] Lustre: server umount lustre-MDT0001 complete [ 7272.697909] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7275.536658] Key type lgssc unregistered [ 7275.813598] LNet: 235786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7275.827651] LNetError: 235786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7275.844734] LNet: Removed LNI 192.168.203.137@tcp [ 7276.812516] Key type .llcrypt unregistered [ 7276.817583] Key type ._llcrypt unregistered [ 7297.807750] Key type ._llcrypt registered [ 7297.811511] Key type .llcrypt registered [ 7297.881409] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7298.868610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7298.896873] alg: No test for adler32 (adler32-zlib) [ 7299.939929] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7300.188532] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7301.840419] Key type lgssc registered [ 7302.738635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7571.601425] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7574.285214] Key type lgssc unregistered [ 7574.564995] LNet: 237804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7574.572850] LNetError: 237804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7574.589659] LNet: Removed LNI 192.168.203.137@tcp [ 7575.483530] Key type .llcrypt unregistered [ 7575.485373] Key type ._llcrypt unregistered [ 7599.350405] Key type ._llcrypt registered [ 7599.352963] Key type .llcrypt registered [ 7599.449525] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 7613.992680] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 7614.858128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7614.975288] alg: No test for adler32 (adler32-zlib) [ 7616.154245] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7616.500894] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 7618.351175] Key type lgssc registered [ 7619.868661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7653.673913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7380 sec ========== 10:56:21 (1788447381) [ 7655.205300] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:56:23 (1788447383) === [ 7658.666199] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:56:26 (1788447386) === [ 7696.677984] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 7699.181412] Key type lgssc unregistered [ 7699.539238] LNet: 243036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7699.546213] LNetError: 243036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7699.558501] LNet: Removed LNI 192.168.203.137@tcp [ 7700.201684] Key type .llcrypt unregistered [ 7700.206531] Key type ._llcrypt unregistered