[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482527517 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002402] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010069] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.022562] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028244] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031120] *** VALIDATE tmpfs *** [ 0.032475] *** VALIDATE proc *** [ 0.033261] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043338] debug: unmapping init [mem 0xffffffff8bc59000-0xffffffff8bc60fff] [ 0.045967] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046723] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054313] rcu: Hierarchical SRCU implementation. [ 0.056489] smp: Bringing up secondary CPUs ... [ 0.057599] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.063035] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147079] node 0 deferred pages initialised in 77ms [ 0.150093] devtmpfs: initialized [ 0.151238] x86/mm: Memory block size: 128MB [ 0.153759] gcov: version magic: 0x41383552 [ 0.157018] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159265] pinctrl core: initialized pinctrl subsystem [ 0.160184] [ 0.160754] ************************************************************* [ 0.161012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162014] ** ** [ 0.163015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164016] ** ** [ 0.165013] ** This means that this kernel is built to expose internal ** [ 0.166015] ** IOMMU data structures, which may compromise security on ** [ 0.167014] ** your system. ** [ 0.168014] ** ** [ 0.169012] ** If you see this message and you are not debugging the ** [ 0.170015] ** kernel, report this immediately to your vendor! ** [ 0.171016] ** ** [ 0.172015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173014] ************************************************************* [ 0.174706] NET: Registered protocol family 16 [ 0.175428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178444] cpuidle: using governor menu [ 0.180813] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183510] PCI: Using configuration type 1 for base access [ 0.185142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202075] cryptd: max_cpu_qlen set to 1000 [ 0.204048] ACPI: Added _OSI(Module Device) [ 0.206019] ACPI: Added _OSI(Processor Device) [ 0.207014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213246] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221412] ACPI: Interpreter enabled [ 0.223058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224013] ACPI: Using IOAPIC for interrupt routing [ 0.226129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242595] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.251076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.256341] acpiphp: Slot [2] registered [ 0.258179] acpiphp: Slot [5] registered [ 0.259169] acpiphp: Slot [6] registered [ 0.261158] acpiphp: Slot [7] registered [ 0.262146] acpiphp: Slot [8] registered [ 0.264149] acpiphp: Slot [9] registered [ 0.266124] acpiphp: Slot [10] registered [ 0.267159] acpiphp: Slot [3] registered [ 0.269113] acpiphp: Slot [4] registered [ 0.270121] acpiphp: Slot [11] registered [ 0.272075] acpiphp: Slot [12] registered [ 0.273086] acpiphp: Slot [13] registered [ 0.274120] acpiphp: Slot [14] registered [ 0.275118] acpiphp: Slot [15] registered [ 0.277111] acpiphp: Slot [16] registered [ 0.278121] acpiphp: Slot [17] registered [ 0.280092] acpiphp: Slot [18] registered [ 0.281095] acpiphp: Slot [19] registered [ 0.282117] acpiphp: Slot [20] registered [ 0.283102] acpiphp: Slot [21] registered [ 0.285125] acpiphp: Slot [22] registered [ 0.286100] acpiphp: Slot [23] registered [ 0.288111] acpiphp: Slot [24] registered [ 0.290105] acpiphp: Slot [25] registered [ 0.291109] acpiphp: Slot [26] registered [ 0.293133] acpiphp: Slot [27] registered [ 0.294103] acpiphp: Slot [28] registered [ 0.296120] acpiphp: Slot [29] registered [ 0.298128] acpiphp: Slot [30] registered [ 0.299104] acpiphp: Slot [31] registered [ 0.301097] PCI host bridge to bus 0000:00 [ 0.303022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322484] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326331] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340632] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351016] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359451] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392053] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.443979] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490106] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533213] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.550014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570039] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.580823] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.589018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.596020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.621910] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.631022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.637016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.650027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.660111] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.662444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.664411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.667386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678127] iommu: Default domain type: Passthrough [ 0.680523] SCSI subsystem initialized [ 0.682159] ACPI: bus type USB registered [ 0.683096] usbcore: registered new interface driver usbfs [ 0.686092] usbcore: registered new interface driver hub [ 0.687062] usbcore: registered new device driver usb [ 0.689176] pps_core: LinuxPPS API ver. 1 registered [ 0.691012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694070] PTP clock support registered [ 0.696111] EDAC MC: Ver: 3.0.0 [ 0.698187] PCI: Using ACPI for IRQ routing [ 0.699944] NetLabel: Initializing [ 0.700009] NetLabel: domain hash size = 128 [ 0.701013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702074] NetLabel: unlabeled traffic allowed by default [ 0.703150] vgaarb: loaded [ 0.705342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.707017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.712014] clocksource: Switched to clocksource kvm-clock [ 0.818965] VFS: Disk quotas dquot_6.6.0 [ 0.820424] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822520] *** VALIDATE ramfs *** [ 0.823887] *** VALIDATE hugetlbfs *** [ 0.825544] pnp: PnP ACPI init [ 0.827989] pnp: PnP ACPI: found 6 devices [ 0.856336] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.859973] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.862423] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.864910] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.867643] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.870375] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.872833] NET: Registered protocol family 2 [ 0.875483] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.881104] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.884747] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.890484] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.894094] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900466] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903505] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.906732] NET: Registered protocol family 1 [ 0.910064] RPC: Registered named UNIX socket transport module. [ 0.912291] RPC: Registered udp transport module. [ 0.914304] RPC: Registered tcp transport module. [ 0.916357] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.919222] NET: Registered protocol family 44 [ 0.920915] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.923115] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.925619] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.928219] PCI: CLS 0 bytes, default 64 [ 0.930223] Unpacking initramfs... [ 2.349550] debug: unmapping init [mem 0xffff8d6d3cc54000-0xffff8d6d3ffbffff] [ 2.353713] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.356218] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.359434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.856749] Initialise system trusted keyrings [ 2.859480] Key type blacklist registered [ 2.861422] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.871807] zbud: loaded [ 2.875567] *** VALIDATE nfs *** [ 2.876629] *** VALIDATE nfs4 *** [ 2.878035] pstore: using deflate compression [ 2.881105] Platform Keyring initialized [ 2.991530] NET: Registered protocol family 38 [ 2.993329] Key type asymmetric registered [ 2.994984] Asymmetric key parser 'x509' registered [ 2.996834] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.000118] io scheduler mq-deadline registered [ 3.001855] io scheduler kyber registered [ 3.003818] io scheduler bfq registered [ 3.005653] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.008350] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.011113] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.014072] ACPI: Power Button [PWRF] [ 3.019186] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026121] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.045561] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.056458] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.074875] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.105191] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.137519] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.143144] Non-volatile memory driver v1.3 [ 3.144528] Linux agpgart interface v0.103 [ 3.172884] virtio_blk virtio1: [vda] 149896 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.175039] vda: detected capacity change from 0 to 76746752 [ 3.189511] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.192234] vdb: detected capacity change from 0 to 1073741824 [ 3.207103] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.210696] vdc: detected capacity change from 0 to 2621440000 [ 3.225247] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.228140] vdd: detected capacity change from 0 to 2621440000 [ 3.244705] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.247985] vde: detected capacity change from 0 to 4294967296 [ 3.262406] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.265561] vdf: detected capacity change from 0 to 4294967296 [ 3.272602] libphy: Fixed MDIO Bus: probed [ 3.277944] usbcore: registered new interface driver usbserial_generic [ 3.280829] usbserial: USB Serial support registered for generic [ 3.283465] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.288742] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.291070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.294148] mousedev: PS/2 mouse device common for all mice [ 3.298032] rtc_cmos 00:05: RTC can wake from S4 [ 3.301744] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.304490] rtc_cmos 00:05: registered as rtc0 [ 3.308329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.308573] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.314147] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.314675] intel_pstate: CPU model not supported [ 3.321190] hid: raw HID events driver (C) Jiri Kosina [ 3.323546] usbcore: registered new interface driver usbhid [ 3.325624] usbhid: USB HID core driver [ 3.327359] drop_monitor: Initializing network drop monitor service [ 3.330114] Initializing XFRM netlink socket [ 3.332326] NET: Registered protocol family 10 [ 3.335725] Segment Routing with IPv6 [ 3.337361] NET: Registered protocol family 17 [ 3.339869] mpls_gso: MPLS GSO support [ 3.346166] RAS: Correctable Errors collector initialized. [ 3.348461] AVX version of gcm_enc/dec engaged. [ 3.350238] AES CTR mode by8 optimization enabled [ 3.431280] sched_clock: Marking stable (3431196008, 0)->(4407269526, -976073518) [ 3.435280] registered taskstats version 1 [ 3.437461] Loading compiled-in X.509 certificates [ 3.439511] zswap: loaded using pool lzo/zbud [ 3.463446] Key type big_key registered [ 3.475568] Key type encrypted registered [ 3.477416] ima: No TPM chip found, activating TPM-bypass! [ 3.480052] ima: Allocated hash algorithm: sha1 [ 3.482181] ima: No architecture policies found [ 3.484486] evm: Initialising EVM extended attributes: [ 3.486889] evm: security.selinux [ 3.488242] evm: security.ima [ 3.490101] evm: security.capability [ 3.491586] evm: HMAC attrs: 0x1 [ 3.493823] rtc_cmos 00:05: setting system clock to 2026-09-03 08:03:31 UTC (1788422611) [ 3.499839] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 3.503278] debug: unmapping init [mem 0xffffffff8b982000-0xffffffff8bc58fff] [ 3.514078] Write protecting the kernel read-only data: 28672k [ 3.518282] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.521683] debug: unmapping init [mem 0xffffffff8a914000-0xffffffff8a9fffff] [ 3.555764] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.564560] systemd[1]: Detected virtualization kvm. [ 3.566855] systemd[1]: Detected architecture x86-64. [ 3.568890] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.592884] systemd[1]: No hostname configured. [ 3.594633] systemd[1]: Set hostname to . [ 3.596618] random: systemd: uninitialized urandom read (16 bytes read) [ 3.599054] systemd[1]: Initializing machine ID from random generator. [ 3.739907] random: systemd: uninitialized urandom read (16 bytes read) [ 3.742492] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.746978] random: systemd: uninitialized urandom read (16 bytes read) [ 3.749623] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.754953] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.392581] device-mapper: uevent: version 1.0.3 [ 4.394951] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.141735] virtio_net virtio0 ens2: renamed from eth0 [ 5.148885] random: fast init done [ 5.301450] scsi host0: ata_piix [ 5.411309] scsi host1: ata_piix [ 5.413673] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.416447] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.088320] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.995486] random: crng init done [ 10.009201] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.487386] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.645133] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.884111] SELinux: Disabled at runtime. [ 11.951888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.962225] systemd[1]: Detected virtualization kvm. [ 11.964240] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.488172] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.490815] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.495078] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.499358] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.503510] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.509923] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.524786] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ 12.659678] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.980733] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.396697] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.413843] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.472253] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.494145] EDAC sbridge: Ver: 1.1.2 [ 15.316842] Key type dns_resolver registered [ 15.619800] NFS: Registering the id_resolver key type [ 15.621841] Key type id_resolver registered [ 15.623120] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg360-server login: [ 50.367729] libcfs: loading out-of-tree module taints kernel. [ 50.398110] Key type ._llcrypt registered [ 50.399803] Key type .llcrypt registered [ 50.590856] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 79.021353] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 81.241501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.287054] alg: No test for adler32 (adler32-zlib) [ 83.485470] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 84.640969] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 86.399212] Key type lgssc registered [ 88.553589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.269367] hrtimer: interrupt took 16659527 ns [ 110.077790] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.419900] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 169.872251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.942258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.421825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.469727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.619495] Lustre: lustre-MDT0000: new disk, initializing [ 171.732584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.743436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.136995] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.442541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.575365] Lustre: 6523:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.656119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.663323] Lustre: Skipped 1 previous similar message [ 190.751518] Lustre: lustre-MDT0001: new disk, initializing [ 190.894242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.934724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.945783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.822299] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.926678] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.863400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.233180] Lustre: lustre-OST0000: new disk, initializing [ 214.237110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.245792] Lustre: 8463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.396084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.752686] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.377178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.406011] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.493327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.118160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.341654] Lustre: lustre-OST0001: new disk, initializing [ 236.347568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 236.356153] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.428815] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.750682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.761459] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.832962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.922524] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.405900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.017684] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.628221] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.469643] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing yml_node [ 280.098205] Lustre: DEBUG MARKER: Client: 2.17.58.7 [ 283.111990] Lustre: DEBUG MARKER: MDS: 2.17.58.7 [ 285.859932] Lustre: DEBUG MARKER: OSS: 2.17.58.7 [ 287.477745] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 04:08:14 EDT 2026 [ 304.522846] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 306.027764] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.070315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.084870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.087767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.112224] Lustre: Skipped 3 previous similar messages [ 329.186343] Lustre: server umount lustre-MDT0000 complete [ 333.727868] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788422941 with bad export cookie 4642232599574434912 [ 333.742564] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.743298] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 334.308649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.311881] LustreError: 8048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.312057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.312061] Lustre: Skipped 3 previous similar messages [ 334.339608] Lustre: Skipped 1 previous similar message [ 334.386883] LustreError: 8048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 338.341392] LustreError: 6533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.342118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.416179] Lustre: Skipped 2 previous similar messages [ 340.484068] Lustre: server umount lustre-MDT0001 complete [ 345.403384] Lustre: server umount lustre-OST0000 complete [ 350.013378] Lustre: server umount lustre-OST0001 complete [ 359.755310] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 375.387824] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 413.503718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.909326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.942414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.076474] Lustre: lustre-MDT0000: new disk, initializing [ 414.287046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.325694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.618033] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 431.303859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.444564] Lustre: 15113:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 431.545147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 431.553799] Lustre: Skipped 1 previous similar message [ 431.674958] Lustre: lustre-MDT0001: new disk, initializing [ 431.870615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.922802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.954200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 436.767119] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 447.755848] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.959717] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.975986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.257394] Lustre: lustre-OST0000: new disk, initializing [ 464.261886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.273570] Lustre: 17084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.401996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.212207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.224332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.371358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.847491] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 480.612887] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.357256] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.594709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.912836] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 491.348889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.128910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.128982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.143667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.173351] Lustre: Skipped 1 previous similar message [ 502.255839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.287908] Lustre: Skipped 1 previous similar message [ 507.871155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.228438] Lustre: server umount lustre-OST0000 complete [ 514.544592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.557811] LustreError: Skipped 1 previous similar message [ 514.564637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.581595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.588682] Lustre: Skipped 3 previous similar messages [ 519.872335] Lustre: server umount lustre-MDT0000 complete [ 522.736869] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.764917] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 524.713190] LustreError: 15106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423132 with bad export cookie 4642232599574436606 [ 524.715050] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.723958] LustreError: 15106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 525.029440] Lustre: server umount lustre-MDT0001 complete [ 528.245598] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:12:15 (1788423135) [ 534.548974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.875529] LustreError: 19349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.933859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.662090] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 540.135053] LustreError: 19350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.684120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.220516] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 559.079148] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 565.833715] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.682553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.179683] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 587.057898] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 597.985616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.985707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 598.000810] Lustre: Skipped 1 previous similar message [ 598.003639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 598.020785] Lustre: Skipped 2 previous similar messages [ 599.635802] Lustre: server umount lustre-OST0000 complete [ 609.565452] Lustre: server umount lustre-MDT0000 complete [ 613.350857] LustreError: 21841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.379179] LustreError: 21841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 613.527781] LustreError: 22130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423221 with bad export cookie 4642232599574437530 [ 613.532279] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.544422] LustreError: 22130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 614.240663] Lustre: server umount lustre-MDT0001 complete [ 626.395317] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 629.256722] Key type lgssc unregistered [ 629.554901] LNet: 22882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.569757] LNetError: 22882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.602059] LNet: Removed LNI 192.168.203.160@tcp [ 630.642227] Key type .llcrypt unregistered [ 630.644112] Key type ._llcrypt unregistered [ 650.500454] Key type ._llcrypt registered [ 650.503589] Key type .llcrypt registered [ 650.661479] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 651.622521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 651.679250] alg: No test for adler32 (adler32-zlib) [ 652.785815] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 653.039355] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 654.718266] Key type lgssc registered [ 655.692065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 663.152837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 663.165340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.591835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.032888] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 674.789872] LustreError: 24271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.815461] LustreError: 24271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 677.808586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.284361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 683.017446] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 690.411199] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 695.634667] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.656093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 702.137525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.956911] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 709.308380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 715.081859] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 730.080324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.085574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.092416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 730.104744] Lustre: Skipped 1 previous similar message [ 730.114613] Lustre: Skipped 1 previous similar message [ 731.850677] Lustre: server umount lustre-OST0000 complete [ 739.808797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 739.829499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.855461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.900988] Lustre: server umount lustre-MDT0000 complete [ 745.442962] LustreError: 24999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.463167] LustreError: 24999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 747.118746] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423355 with bad export cookie 11883782872145938884 [ 747.128306] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.130284] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 747.409952] Lustre: server umount lustre-MDT0001 complete [ 756.282595] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 758.628963] Key type lgssc unregistered [ 758.859185] LNet: 28246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.869896] LNetError: 28246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.893092] LNet: Removed LNI 192.168.203.160@tcp [ 759.621658] Key type .llcrypt unregistered [ 759.624745] Key type ._llcrypt unregistered [ 779.528981] Key type ._llcrypt registered [ 779.532570] Key type .llcrypt registered [ 779.598557] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 780.747259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.769242] alg: No test for adler32 (adler32-zlib) [ 781.847413] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 782.107109] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 783.759288] Key type lgssc registered [ 785.017258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.798304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 793.856593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.451726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.685169] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 805.856338] LustreError: 29635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.871699] LustreError: 29635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 809.965593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.593660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.602651] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 825.865556] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 831.045778] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 845.538748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.115355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.630462] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 865.607932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 868.140339] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.045446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.051138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.072540] Lustre: Skipped 1 previous similar message [ 880.085235] Lustre: Skipped 1 previous similar message [ 883.854353] Lustre: server umount lustre-OST0000 complete [ 890.338600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.355416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.358089] Lustre: Skipped 1 previous similar message [ 890.378211] Lustre: Skipped 1 previous similar message [ 894.653263] Lustre: server umount lustre-MDT0000 complete [ 895.473317] LustreError: 29632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.499528] LustreError: 29632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 898.278823] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423506 with bad export cookie 13340848605261483505 [ 898.279660] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.290060] LustreError: 29616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 898.732589] Lustre: server umount lustre-MDT0001 complete [ 910.996599] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 915.316610] Key type lgssc unregistered [ 915.782473] LNet: 33517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.803896] LNetError: 33517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.820878] LNet: Removed LNI 192.168.203.160@tcp [ 917.065874] Key type .llcrypt unregistered [ 917.071417] Key type ._llcrypt unregistered [ 944.072737] Key type ._llcrypt registered [ 944.079769] Key type .llcrypt registered [ 944.185035] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 945.064045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.207172] alg: No test for adler32 (adler32-zlib) [ 946.318626] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 946.479205] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 948.199163] Key type lgssc registered [ 949.326141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.584225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.601548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 959.341299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.267440] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 969.712336] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.745842] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 974.816889] LustreError: 34925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.989389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.451044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.090535] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 986.816749] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.805601] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1001.401932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.228474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.283058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1009.059897] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1017.403464] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.137448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1031.144627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.156646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.166340] Lustre: Skipped 1 previous similar message [ 1036.258903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.267691] Lustre: Skipped 2 previous similar messages [ 1036.672341] Lustre: server umount lustre-OST0000 complete [ 1041.383099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.391573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1041.407971] Lustre: Skipped 1 previous similar message [ 1041.418704] Lustre: Skipped 1 previous similar message [ 1046.498062] LustreError: 34925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.518701] LustreError: 34925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1046.545814] Lustre: server umount lustre-MDT0000 complete [ 1050.192798] LustreError: 34904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423658 with bad export cookie 11627583055798969283 [ 1050.194938] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.204729] LustreError: 34904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1050.596698] Lustre: server umount lustre-MDT0001 complete [ 1060.679883] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1063.314857] Key type lgssc unregistered [ 1063.656861] LNet: 38481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.662474] LNetError: 38481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.671952] LNet: Removed LNI 192.168.203.160@tcp [ 1064.477623] Key type .llcrypt unregistered [ 1064.481986] Key type ._llcrypt unregistered [ 1085.830758] Key type ._llcrypt registered [ 1085.834273] Key type .llcrypt registered [ 1085.947612] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1086.928238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.108878] alg: No test for adler32 (adler32-zlib) [ 1088.182786] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1088.488909] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1090.159392] Key type lgssc registered [ 1091.489593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.431841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.448111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.934932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.162909] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1110.505821] LustreError: 39891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.552872] LustreError: 39891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1115.607612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.625087] LustreError: 39890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.372232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.406642] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1130.710908] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1135.968899] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.402451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.879357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.980749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1148.681733] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1156.170211] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1166.819674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1166.820819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.821722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1166.857943] Lustre: Skipped 1 previous similar message [ 1168.326762] Lustre: server umount lustre-OST0000 complete [ 1191.396810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.402431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.416821] Lustre: Skipped 1 previous similar message [ 1191.437494] Lustre: Skipped 2 previous similar messages [ 1196.515183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.524330] Lustre: Skipped 1 previous similar message [ 1201.634575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.644988] Lustre: Skipped 1 previous similar message [ 1203.170017] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.285539] Lustre: server umount lustre-MDT0000 complete [ 1206.564197] LustreError: 39870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423814 with bad export cookie 13127298104703709071 [ 1206.571872] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.883466] Lustre: server umount lustre-MDT0001 complete [ 1224.116195] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1227.118487] Key type lgssc unregistered [ 1227.548689] LNet: 43598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.570485] LNetError: 43598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.585703] LNet: Removed LNI 192.168.203.160@tcp [ 1228.396527] Key type .llcrypt unregistered [ 1228.398810] Key type ._llcrypt unregistered [ 1254.594459] Key type ._llcrypt registered [ 1254.597852] Key type .llcrypt registered [ 1254.782462] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1255.795685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.812277] alg: No test for adler32 (adler32-zlib) [ 1257.490371] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1258.112405] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1259.935322] Key type lgssc registered [ 1261.802301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.965718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1274.051106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.088126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.907253] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1286.639566] LustreError: 45008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.671213] LustreError: 45008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1291.748335] LustreError: 45007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.346175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.993962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1300.033875] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1308.167492] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1314.580359] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.575697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1321.953279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.012573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1328.430483] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1338.865557] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1349.602803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.610817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.630934] Lustre: Skipped 1 previous similar message [ 1349.660982] Lustre: Skipped 2 previous similar messages [ 1351.063586] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1353.147149] Lustre: server umount lustre-MDT0000 complete [ 1354.730943] LustreError: 45003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.772537] LustreError: 45003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1357.996737] LustreError: 44989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423966 with bad export cookie 13073342482926686723 [ 1358.017765] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.019080] LustreError: 44989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1359.839956] LustreError: 45001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.841872] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.903266] LustreError: 45001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1359.934869] Lustre: Skipped 1 previous similar message [ 1359.951319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1363.872144] LustreError: 45003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.879365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1363.881737] LustreError: 45003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1363.887694] Lustre: Skipped 1 previous similar message [ 1364.429269] Lustre: server umount lustre-MDT0001 complete [ 1379.515454] Lustre: server umount lustre-OST0000 complete [ 1395.016488] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1397.750558] Key type lgssc unregistered [ 1398.006649] LNet: 48869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.018456] LNetError: 48869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.052386] LNet: Removed LNI 192.168.203.160@tcp [ 1398.898570] Key type .llcrypt unregistered [ 1398.901267] Key type ._llcrypt unregistered [ 1422.508925] Key type ._llcrypt registered [ 1422.513895] Key type .llcrypt registered [ 1422.662460] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1423.766905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.815939] alg: No test for adler32 (adler32-zlib) [ 1424.829931] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1425.115391] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1426.897448] Key type lgssc registered [ 1428.366534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.100572] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.117188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.679661] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1456.799522] LustreError: 50251:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 1483.487230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.755420] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1495.201580] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1664.894815] Lustre: server umount lustre-OST0000 complete [ 1684.666645] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1688.214025] Key type lgssc unregistered [ 1688.680707] LNet: 51926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.690163] LNetError: 51926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.713290] LNet: Removed LNI 192.168.203.160@tcp [ 1690.089567] Key type .llcrypt unregistered [ 1690.098201] Key type ._llcrypt unregistered [ 1724.335252] Key type ._llcrypt registered [ 1724.339892] Key type .llcrypt registered [ 1724.504376] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1725.167795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.183677] alg: No test for adler32 (adler32-zlib) [ 1726.210543] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1726.406563] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1728.089101] Key type lgssc registered [ 1728.960821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.230141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.263328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.733703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.343802] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1747.947729] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1747.974283] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1751.546489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.953319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1755.793393] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1763.181059] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1768.679956] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1776.625024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.923748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.075065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1784.246972] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1791.888263] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.129508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.136310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.187761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.228663] Lustre: Skipped 2 previous similar messages [ 1806.309276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.316823] Lustre: Skipped 1 previous similar message [ 1808.352748] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1808.592707] Lustre: server umount lustre-OST0000 complete [ 1813.493731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1813.512683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.545718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.557228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.737339] Lustre: server umount lustre-MDT0000 complete [ 1821.680633] LustreError: 53338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.697503] LustreError: 53338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1822.420358] LustreError: 53318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424430 with bad export cookie 3489149055457917048 [ 1822.422603] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1822.432394] LustreError: 53318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1822.808828] Lustre: server umount lustre-MDT0001 complete [ 1834.416875] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1837.185925] Key type lgssc unregistered [ 1837.495561] LNet: 56868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.506483] LNetError: 56868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.519187] LNet: Removed LNI 192.168.203.160@tcp [ 1838.423592] Key type .llcrypt unregistered [ 1838.426836] Key type ._llcrypt unregistered [ 1859.717766] Key type ._llcrypt registered [ 1859.722201] Key type .llcrypt registered [ 1859.804570] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1860.698221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.799896] alg: No test for adler32 (adler32-zlib) [ 1861.882698] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1862.179840] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1863.839300] Key type lgssc registered [ 1865.093334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.034580] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.067274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.201694] Lustre: 58242:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1889.759333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1894.879311] LustreError: 58230:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 1899.488913] Lustre: 58245:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1905.183334] Lustre: 58230:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 1915.499726] Lustre: 58242:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1918.367314] Lustre: 58247:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 1920.991740] Lustre: 58245:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1925.090296] Lustre: 58242:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1925.727226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.607531] LustreError: 58247:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 1933.132945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1935.334187] Lustre: 58242:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1935.370902] Lustre: 58242:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 1941.626388] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1949.094208] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1949.323319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.029294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.100207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1954.836917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1955.304116] LustreError: 58260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.324774] LustreError: 58260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1960.418862] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.433927] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1964.270228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.530370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.322584] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1976.822400] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1982.512223] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.006430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.008884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.020654] Lustre: Skipped 1 previous similar message [ 1985.021732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1985.026532] Lustre: Skipped 1 previous similar message [ 1990.117932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.133720] Lustre: Skipped 1 previous similar message [ 1990.707561] Lustre: server umount lustre-OST0000 complete [ 2015.714770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2015.721452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.739619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.746372] Lustre: Skipped 1 previous similar message [ 2020.115283] Lustre: server umount lustre-MDT0000 complete [ 2020.834119] LustreError: 59281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.869203] LustreError: 59281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2023.929176] LustreError: 58236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424631 with bad export cookie 17694012940661860123 [ 2023.945854] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.947200] LustreError: 58236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2038.239234] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2038.426371] Lustre: server umount lustre-MDT0001 complete [ 2049.429395] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2054.437242] Key type lgssc unregistered [ 2054.957799] LNet: 61989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.971683] LNetError: 61989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.009915] LNet: Removed LNI 192.168.203.160@tcp [ 2056.716753] Key type .llcrypt unregistered [ 2056.720091] Key type ._llcrypt unregistered [ 2095.222861] Key type ._llcrypt registered [ 2095.225888] Key type .llcrypt registered [ 2095.396605] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2096.593089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.761440] alg: No test for adler32 (adler32-zlib) [ 2097.844292] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2098.214897] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2099.871290] Key type lgssc registered [ 2101.122839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2108.707175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2108.723138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.378386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.483312] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2120.679568] LustreError: 63399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.717032] LustreError: 63399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2125.087081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.752099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.683561] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2142.938870] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2148.444438] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.232531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.514267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.743553] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2168.815839] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2180.106758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2190.311286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.314194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2190.325849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2190.330226] Lustre: Skipped 1 previous similar message [ 2190.362837] Lustre: Skipped 1 previous similar message [ 2193.636265] Lustre: server umount lustre-OST0000 complete [ 2200.544816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.572537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.579798] Lustre: Skipped 1 previous similar message [ 2203.742500] Lustre: server umount lustre-MDT0000 complete [ 2205.672261] LustreError: 64127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.708410] LustreError: 64127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2207.305908] LustreError: 63379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424815 with bad export cookie 18182716081603210383 [ 2207.315292] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.316110] LustreError: 63379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2207.694445] Lustre: server umount lustre-MDT0001 complete [ 2218.355970] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2221.104632] Key type lgssc unregistered [ 2221.400767] LNet: 66955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.416210] LNetError: 66955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.449350] LNet: Removed LNI 192.168.203.160@tcp [ 2222.545493] Key type .llcrypt unregistered [ 2222.549872] Key type ._llcrypt unregistered [ 2250.504756] Key type ._llcrypt registered [ 2250.510301] Key type .llcrypt registered [ 2250.741276] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2252.326642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.390270] alg: No test for adler32 (adler32-zlib) [ 2253.475501] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2253.771520] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2255.495257] Key type lgssc registered [ 2257.084397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2265.133054] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:41:12 (1788424872) [ 2271.093456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.125478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.508780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.592608] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2282.980445] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.007516] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2284.157576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.592341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2288.588590] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2295.772117] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2301.311414] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2307.327322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2307.637627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.960192] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2317.867684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2320.486598] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2327.165882] Lustre: Failing over lustre-MDT0000 [ 2327.498263] Lustre: server umount lustre-MDT0000 complete [ 2328.035832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.042974] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.059975] Lustre: Skipped 2 previous similar messages [ 2328.099991] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2333.153954] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.178613] LustreError: 68652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2335.261241] LustreError: 70499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424943 with bad export cookie 10199387848029064797 [ 2335.262516] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.274284] LustreError: 70499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2335.303422] LustreError: 68657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.307012] LustreError: 67723:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 2335.310922] Lustre: 71608:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2335.324969] LustreError: 68657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2337.438961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.741311] Lustre: *** cfs_fail_loc=135, val=0*** [ 2337.795745] LustreError: 71748:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2337.802158] LustreError: 71748:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.203.160@tcp: cfg command failed: rc = -2 [ 2337.808820] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2337.808820] [ 2337.816141] LustreError: mgc192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2337.834097] LustreError: 71735:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2337.844066] LustreError: 71735:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 2337.855561] LustreError: 71735:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 2337.862144] LustreError: 71735:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 2337.885695] Lustre: server umount lustre-MDT0000 complete [ 2337.890989] LustreError: 71735:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2340.327847] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.350631] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2344.432876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.516187] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.810924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.865232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2348.854521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2349.246265] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2350.074943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2350.108441] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2350.155616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2360.297954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.326474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2363.614307] Lustre: server umount lustre-OST0000 complete [ 2370.529444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.533747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.534078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.534083] Lustre: Skipped 1 previous similar message [ 2370.562148] Lustre: Skipped 2 previous similar messages [ 2375.650297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.660965] Lustre: Skipped 2 previous similar messages [ 2380.773244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.782866] Lustre: Skipped 1 previous similar message [ 2382.305442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2382.556082] Lustre: server umount lustre-MDT0000 complete [ 2385.889105] LustreError: 68653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.902371] LustreError: 68653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2386.497939] LustreError: 68638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424994 with bad export cookie 10199387848029066113 [ 2386.510698] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.525728] LustreError: 68638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2387.025951] Lustre: server umount lustre-MDT0001 complete [ 2397.542532] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2400.656877] Key type lgssc unregistered [ 2400.901804] LNet: 73530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.912509] LNetError: 73530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.931264] LNet: Removed LNI 192.168.203.160@tcp [ 2401.910880] Key type .llcrypt unregistered [ 2401.913811] Key type ._llcrypt unregistered [ 2422.332579] Key type ._llcrypt registered [ 2422.334479] Key type .llcrypt registered [ 2422.416357] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2423.369801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.403851] alg: No test for adler32 (adler32-zlib) [ 2424.480795] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2424.713327] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2426.421261] Key type lgssc registered [ 2427.578752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.321391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.346873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.845172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.634111] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2446.337171] LustreError: 74920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.366248] LustreError: 74920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2450.201239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.700828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2454.705330] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2461.647919] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2467.451428] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.129965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.656919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.801726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2481.848781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2489.151690] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2495.676645] Lustre: Failing over lustre-MDT0000 [ 2496.057717] Lustre: server umount lustre-MDT0000 complete [ 2496.917227] LustreError: 76160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.007040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2497.028846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.005506] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.022149] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2506.259203] LustreError: mgc192.168.203.160@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2506.261232] LustreError: 74899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425114 with bad export cookie 12454108891719660350 [ 2506.270271] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.271127] Lustre: 74911:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 2506.273519] Lustre: 77971:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2506.273769] LustreError: Skipped 1 previous similar message [ 2506.690077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.915240] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2507.028600] LustreError: 78011:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2507.037433] LustreError: 78011:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.203.160@tcp: cfg command failed: rc = -9 [ 2507.044426] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2507.044426] [ 2507.053175] LustreError: mgc192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2507.069480] LustreError: 77999:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2507.079703] LustreError: 77999:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -9 [ 2507.086646] LustreError: 77999:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 2507.092080] LustreError: 77999:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 2507.118651] Lustre: server umount lustre-MDT0000 complete [ 2507.123524] LustreError: 77999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2507.138056] LustreError: 74915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.138070] LustreError: 74915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2512.213060] LustreError: 74915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.247982] LustreError: 74915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2514.886052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.206763] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.217836] LustreError: 78359:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mgc192.168.203.160@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000002f0bf619) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2515.674469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.763168] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2517.370084] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2520.997129] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2521.074688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2521.130061] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2521.184601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2531.302885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.312653] Lustre: Skipped 2 previous similar messages [ 2531.317966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.323682] Lustre: Skipped 1 previous similar message [ 2535.540965] Lustre: server umount lustre-OST0000 complete [ 2541.539745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.541873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.553360] Lustre: Skipped 2 previous similar messages [ 2541.573113] Lustre: Skipped 1 previous similar message [ 2545.606385] Lustre: server umount lustre-MDT0000 complete [ 2546.661252] LustreError: 74920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.684386] LustreError: 74920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2549.650044] LustreError: 74899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425157 with bad export cookie 12454108891719661596 [ 2549.664418] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.675381] LustreError: 74899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2550.210312] Lustre: server umount lustre-MDT0001 complete [ 2561.422229] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2564.478752] Key type lgssc unregistered [ 2564.812704] LNet: 79792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.826181] LNetError: 79792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2564.857707] LNet: Removed LNI 192.168.203.160@tcp [ 2565.840298] Key type .llcrypt unregistered [ 2565.842602] Key type ._llcrypt unregistered [ 2578.025985] Key type ._llcrypt registered [ 2578.028882] Key type .llcrypt registered [ 2578.132221] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2578.483290] Key type .llcrypt unregistered [ 2578.487469] Key type ._llcrypt unregistered [ 2600.615460] Key type ._llcrypt registered [ 2600.617822] Key type .llcrypt registered [ 2600.711329] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2601.482023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2601.510393] alg: No test for adler32 (adler32-zlib) [ 2602.774914] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2602.977955] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2604.647274] Key type lgssc registered [ 2605.978654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2614.138835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2614.166378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.633667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.106520] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2626.021867] LustreError: 81686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.048217] LustreError: 81686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2628.853619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.538532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.586308] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2644.425826] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2650.539913] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2657.780841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.467976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.579741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2666.096372] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2673.685702] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.413993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.414234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2685.441694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.446582] Lustre: Skipped 1 previous similar message [ 2689.388824] Lustre: server umount lustre-OST0000 complete [ 2695.651887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.668942] Lustre: Skipped 1 previous similar message [ 2695.674049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.686051] Lustre: Skipped 1 previous similar message [ 2699.724443] Lustre: server umount lustre-MDT0000 complete [ 2700.768820] LustreError: 81687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.805901] LustreError: 81687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2704.190775] LustreError: 81668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425312 with bad export cookie 3878569876765145958 [ 2704.195221] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.204812] LustreError: 81668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2704.742449] Lustre: server umount lustre-MDT0001 complete [ 2716.440824] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2719.541433] Key type lgssc unregistered [ 2719.850748] LNet: 85220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.857248] LNetError: 85220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.880637] LNet: Removed LNI 192.168.203.160@tcp [ 2720.827978] Key type .llcrypt unregistered [ 2720.833176] Key type ._llcrypt unregistered [ 2744.272619] Key type ._llcrypt registered [ 2744.275563] Key type .llcrypt registered [ 2744.397295] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2745.711309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.734766] alg: No test for adler32 (adler32-zlib) [ 2746.856543] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2747.135101] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2748.847453] Key type lgssc registered [ 2750.400645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.845405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2758.884725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.524338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.329872] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2770.927671] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.973660] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2773.976919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.639379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.908551] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2786.574580] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2792.795925] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2800.038864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.437532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.557650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2807.023972] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2815.457099] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2825.188333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2825.190617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.196360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.241913] Lustre: Skipped 1 previous similar message [ 2827.176249] Lustre: server umount lustre-OST0000 complete [ 2830.817897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2830.828358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2830.848814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2830.851704] Lustre: Skipped 1 previous similar message [ 2835.424798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2835.428033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.059195] Lustre: server umount lustre-MDT0000 complete [ 2840.546545] LustreError: 86631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.583236] LustreError: 86631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2842.603953] LustreError: 86612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425450 with bad export cookie 14067713783174524468 [ 2842.613820] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2843.054529] Lustre: server umount lustre-MDT0001 complete [ 2854.045870] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2856.956110] Key type lgssc unregistered [ 2857.311663] LNet: 90160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2857.330387] LNetError: 90160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2858.410144] LNet: Removed LNI 192.168.203.160@tcp [ 2859.436166] Key type .llcrypt unregistered [ 2859.438391] Key type ._llcrypt unregistered [ 2884.099660] Key type ._llcrypt registered [ 2884.106310] Key type .llcrypt registered [ 2884.253760] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2885.323493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2885.485974] alg: No test for adler32 (adler32-zlib) [ 2886.547816] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2886.878900] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2888.599142] Key type lgssc registered [ 2889.833708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.687112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2898.733293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.585849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.523872] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2910.690555] LustreError: 91571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.706758] LustreError: 91571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2914.187301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.647107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.408301] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2929.837349] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2937.771817] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2944.983828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2945.369769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.426771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2951.507890] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2958.979215] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2972.134342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.141988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.148927] LustreError: Skipped 1 previous similar message [ 2972.171647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.202305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.203526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.971738] Lustre: server umount lustre-OST0000 complete [ 2985.446365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.452377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2985.470464] Lustre: Skipped 1 previous similar message [ 2985.475090] Lustre: Skipped 2 previous similar messages [ 2989.789317] Lustre: server umount lustre-MDT0000 complete [ 2990.576555] LustreError: 91571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.613757] LustreError: 91571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2994.286837] LustreError: 94376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425602 with bad export cookie 2885442555677241422 [ 2994.296945] LustreError: 94376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2994.297947] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2994.699366] Lustre: server umount lustre-MDT0001 complete [ 3007.767174] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3010.929532] Key type lgssc unregistered [ 3011.255376] LNet: 95128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.264887] LNetError: 95128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.287644] LNet: Removed LNI 192.168.203.160@tcp [ 3012.157888] Key type .llcrypt unregistered [ 3012.161713] Key type ._llcrypt unregistered [ 3036.604354] Key type ._llcrypt registered [ 3036.611734] Key type .llcrypt registered [ 3036.746219] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3038.154769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3038.272050] alg: No test for adler32 (adler32-zlib) [ 3039.428770] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3039.669288] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3041.376840] Key type lgssc registered [ 3042.942775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.224900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.239174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.346438] Lustre: 96524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3067.810467] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3072.927367] LustreError: 96511:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 3077.601344] Lustre: 96526:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3083.233733] Lustre: 96511:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3093.482162] Lustre: 96524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3093.919306] Lustre: 96528:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3098.595716] Lustre: 96526:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3103.711800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.716135] Lustre: 96524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3104.160398] LustreError: 96528:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 3109.732621] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3113.952103] Lustre: 96524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3113.980797] Lustre: 96524:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 3117.134731] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3124.639374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3124.781583] Lustre: server umount lustre-OST0000 complete [ 3133.199437] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:55:40 (1788425740) [ 3140.121230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.566873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.044288] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3145.708744] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.720885] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3150.824405] LustreError: 98159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.094327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.631690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3160.462065] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3169.042642] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3174.756549] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3181.859789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.194189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.271404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3188.595525] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3195.866464] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3211.244033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.248722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.256072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.258537] LustreError: Skipped 1 previous similar message [ 3211.270203] Lustre: Skipped 1 previous similar message [ 3215.701936] Lustre: server umount lustre-OST0000 complete [ 3221.472443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.478085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3221.495452] Lustre: Skipped 2 previous similar messages [ 3221.519028] Lustre: Skipped 1 previous similar message [ 3225.378902] Lustre: server umount lustre-MDT0000 complete [ 3226.594485] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.617185] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3229.069407] LustreError: 98139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425837 with bad export cookie 1661373720035292536 [ 3229.071966] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.085793] LustreError: 98139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3229.403364] Lustre: server umount lustre-MDT0001 complete [ 3240.407847] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3242.983936] Key type lgssc unregistered [ 3243.313771] LNet: 101850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3243.331548] LNetError: 101850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.390713] LNet: Removed LNI 192.168.203.160@tcp [ 3245.323398] Key type .llcrypt unregistered [ 3245.326105] Key type ._llcrypt unregistered [ 3288.790344] Key type ._llcrypt registered [ 3288.792927] Key type .llcrypt registered [ 3288.900486] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3289.799970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3289.897054] alg: No test for adler32 (adler32-zlib) [ 3290.963607] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3291.212612] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3292.879503] Key type lgssc registered [ 3293.979877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3301.171430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3301.195336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.623052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.097799] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3312.112988] LustreError: 104038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.132636] LustreError: 104038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3313.430929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.718041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3317.463864] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3323.866790] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3329.034543] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3335.128367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.538072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.383325] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3347.824942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3349.473362] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3358.180702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.191599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.197996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3358.222953] Lustre: Skipped 1 previous similar message [ 3363.309214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.315300] Lustre: Skipped 2 previous similar messages [ 3364.068973] Lustre: server umount lustre-OST0000 complete [ 3368.420917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.450680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.460945] Lustre: Skipped 1 previous similar message [ 3373.536749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.545200] Lustre: Skipped 1 previous similar message [ 3378.658431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.239192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.472604] Lustre: server umount lustre-MDT0000 complete [ 3383.783163] LustreError: 104766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.793032] LustreError: 104766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3386.609774] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425994 with bad export cookie 8731060041777201597 [ 3386.620783] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.625791] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3387.163502] Lustre: server umount lustre-MDT0001 complete [ 3398.324938] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3401.123613] Key type lgssc unregistered [ 3401.420710] LNet: 107571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3401.431304] LNetError: 107571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3401.464417] LNet: Removed LNI 192.168.203.160@tcp [ 3402.212217] Key type .llcrypt unregistered [ 3402.217590] Key type ._llcrypt unregistered [ 3420.144346] Key type ._llcrypt registered [ 3420.148496] Key type .llcrypt registered [ 3420.267045] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3421.182434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3421.193055] alg: No test for adler32 (adler32-zlib) [ 3422.162941] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3422.476848] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3424.137568] Key type lgssc registered [ 3424.984794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3431.854119] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3431.876180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3432.986417] Lustre: 108824:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3448.479317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3453.600058] LustreError: 108811:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 3458.016099] Lustre: 108826:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3463.839622] Lustre: 108811:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3474.085124] Lustre: 108824:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3475.295298] Lustre: 108829:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3479.521746] Lustre: 108826:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3484.383211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.641591] Lustre: 108824:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3485.537246] LustreError: 108829:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 3489.851264] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3494.887956] Lustre: 108824:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3494.906115] Lustre: 108824:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 3496.012187] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3501.597411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.755430] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3501.922203] LustreError: mgc192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3501.933334] LustreError: 109855:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3501.943029] LustreError: 109855:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 3501.953797] LustreError: 109855:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 3502.043467] Lustre: server umount lustre-MDT0000 complete [ 3502.046268] LustreError: 109855:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3520.063283] Lustre: server umount lustre-OST0000 complete [ 3529.093945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 3537.829295] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3570.675363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.922187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3570.964406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3571.037179] Lustre: lustre-MDT0000: new disk, initializing [ 3571.100473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.123932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3575.809246] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3586.349775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.561874] Lustre: 112649:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3586.669684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3586.674702] Lustre: Skipped 1 previous similar message [ 3586.764078] Lustre: lustre-MDT0001: new disk, initializing [ 3586.926225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3586.995365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3587.030647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3592.389457] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3601.816228] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3607.058312] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3612.818891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3613.063129] Lustre: lustre-OST0000: new disk, initializing [ 3613.070339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3613.078522] Lustre: 114617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3613.162021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.121982] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3619.490976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3619.502618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3619.609223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3627.693644] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3631.824469] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3632.049271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3636.912644] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3637.178741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3640.288481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.298403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.306868] LustreError: Skipped 1 previous similar message [ 3640.335605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3645.495671] Lustre: server umount lustre-OST0000 complete [ 3650.530088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.566056] Lustre: Skipped 1 previous similar message [ 3650.596264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3650.599240] Lustre: Skipped 1 previous similar message [ 3655.649697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.662884] Lustre: Skipped 1 previous similar message [ 3660.771738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.779795] Lustre: Skipped 1 previous similar message [ 3664.351182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3664.664359] Lustre: server umount lustre-MDT0000 complete [ 3665.889569] LustreError: 114912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.907341] LustreError: 114912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3668.182375] LustreError: 112639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426276 with bad export cookie 2467678728326977980 [ 3668.185472] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.193992] LustreError: 112639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3668.735082] Lustre: server umount lustre-MDT0001 complete [ 3676.739541] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:04:44 (1788426284) [ 3679.981594] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3700.389909] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 3710.489466] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3749.641187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.996151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3750.056415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3750.196577] Lustre: lustre-MDT0000: new disk, initializing [ 3750.349462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.375087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3755.787301] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3768.076658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.232385] Lustre: 119518:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3768.495361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3768.504056] Lustre: Skipped 1 previous similar message [ 3768.639602] Lustre: lustre-MDT0001: new disk, initializing [ 3768.794849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.867182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3768.894928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3775.310634] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3787.183968] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3793.860685] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3800.252027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.493923] Lustre: lustre-OST0000: new disk, initializing [ 3800.497907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3800.504295] Lustre: 121490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3800.592784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.838383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3806.853224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3806.870301] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3806.957743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3816.577948] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3821.482096] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3821.756978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3827.380106] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3827.640829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3832.808681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.809344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.815936] LustreError: Skipped 1 previous similar message [ 3832.820055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.829234] Lustre: Skipped 2 previous similar messages [ 3832.843114] Lustre: Skipped 1 previous similar message [ 3836.635206] Lustre: server umount lustre-OST0000 complete [ 3843.042100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.046209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.074881] Lustre: Skipped 1 previous similar message [ 3855.327114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3855.500585] Lustre: server umount lustre-MDT0000 complete [ 3857.376513] LustreError: 119531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.407727] LustreError: 119531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3859.226351] LustreError: 119511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426467 with bad export cookie 2467678728326978904 [ 3859.228633] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.232391] LustreError: 119511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3859.663610] Lustre: server umount lustre-MDT0001 complete [ 3868.843951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.530748] LustreError: 123709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.627465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.910687] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3874.790363] LustreError: 123710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.906554] LustreError: 123709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.541380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.869167] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3910.254605] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3916.768643] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3924.650782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.403749] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3941.438409] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3948.522563] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3953.120290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.126794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.136701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.140220] Lustre: Skipped 5 previous similar messages [ 3955.168488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.606207] Lustre: server umount lustre-OST0000 complete [ 3964.903871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.916170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.938819] Lustre: Skipped 1 previous similar message [ 3978.207241] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3978.393969] Lustre: server umount lustre-MDT0000 complete [ 3980.777775] LustreError: 123709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.809933] LustreError: 123709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3982.871865] LustreError: 123690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426590 with bad export cookie 2467678728326979828 [ 3982.891370] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.383830] Lustre: server umount lustre-MDT0001 complete [ 3996.414831] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4000.137824] Key type lgssc unregistered [ 4000.483620] LNet: 127339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4000.489913] LNetError: 127339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4000.515489] LNet: Removed LNI 192.168.203.160@tcp [ 4001.834898] Key type .llcrypt unregistered [ 4001.837933] Key type ._llcrypt unregistered [ 4022.863625] Key type ._llcrypt registered [ 4022.865351] Key type .llcrypt registered [ 4022.990929] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 4038.644478] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4039.835182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4039.889336] alg: No test for adler32 (adler32-zlib) [ 4041.232590] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 4041.595988] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4043.271167] Key type lgssc registered [ 4044.906130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4087.323584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4087.346050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.676178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4088.705878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4088.810891] Lustre: lustre-MDT0000: new disk, initializing [ 4088.867561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.893985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4093.457539] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4103.605834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.736286] Lustre: 130488:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4103.916835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4103.924516] Lustre: Skipped 1 previous similar message [ 4103.998333] Lustre: lustre-MDT0001: new disk, initializing [ 4104.092176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.183888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4104.204166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4108.766258] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4118.080487] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4123.721589] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4130.808054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4131.105317] Lustre: lustre-OST0000: new disk, initializing [ 4131.109322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4131.122437] Lustre: 132456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4131.245633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.736647] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4141.717669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4141.732746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4141.882942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4149.272528] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4154.148588] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4154.359351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4159.166506] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4159.406135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4162.533632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.534400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.546862] LustreError: Skipped 1 previous similar message [ 4162.561264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.610614] Lustre: Skipped 1 previous similar message [ 4167.663986] Lustre: server umount lustre-OST0000 complete [ 4172.046660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.053643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.063289] Lustre: Skipped 1 previous similar message [ 4172.086075] Lustre: Skipped 2 previous similar messages [ 4177.378278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.394680] Lustre: Skipped 1 previous similar message [ 4178.471516] Lustre: server umount lustre-MDT0000 complete [ 4182.162981] LustreError: 130479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426790 with bad export cookie 15939525955540814542 [ 4182.171151] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4182.185841] LustreError: 130479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4182.589648] Lustre: server umount lustre-MDT0001 complete [ 4191.395026] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:13:18 (1788426798) [ 4198.254267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.839744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.586631] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4204.003321] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.023479] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4208.098043] LustreError: 134869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.139927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.610203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4216.656176] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4224.971038] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.772345] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.209100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.212662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.227365] Lustre: Skipped 1 previous similar message [ 4239.835780] Lustre: server umount lustre-MDT0000 complete [ 4242.400602] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.428391] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4244.581959] LustreError: 134848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426852 with bad export cookie 15939525955540815473 [ 4244.589114] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.593749] LustreError: 134848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4244.974614] Lustre: server umount lustre-MDT0001 complete [ 4257.746963] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:14:24 (1788426864) [ 4265.632574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4265.735463] Lustre: 137297:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4271.071841] Lustre: 137299:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4276.192016] Lustre: 137297:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4281.183769] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4281.311968] Lustre: 137299:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4286.303450] LustreError: 137284:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 4286.433376] Lustre: 137297:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4296.608969] Lustre: 137284:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4296.676546] Lustre: 137297:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4296.683102] Lustre: 137297:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 4308.513435] Lustre: 137301:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4316.128296] Lustre: 137298:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4316.146604] Lustre: 137298:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 3 previous similar messages [ 4317.151327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.820209] LustreError: 137301:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4323.658486] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4331.247129] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4339.295811] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4339.533321] Lustre: server umount lustre-OST0000 complete [ 4348.313619] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:15:55 (1788426955) [ 4355.640800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.930123] LustreError: 138739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.039590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.205688] LustreError: 138740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.249542] LustreError: 138740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4362.626056] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4370.406721] LustreError: 138739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.434355] LustreError: 138739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4371.861919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.950221] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4382.787387] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4387.776587] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4395.671409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.032847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.050891] Lustre: Skipped 1 previous similar message [ 4402.560869] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4409.722200] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4418.534963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4418.540918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.556183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.567862] Lustre: Skipped 3 previous similar messages [ 4423.368164] Lustre: server umount lustre-MDT0000 complete [ 4425.700429] LustreError: 141036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.737661] LustreError: 141036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4427.683570] LustreError: 138720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427035 with bad export cookie 15939525955540816229 [ 4427.691472] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.700307] LustreError: 138720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4428.234458] Lustre: server umount lustre-MDT0001 complete [ 4443.185446] Lustre: server umount lustre-OST0000 complete [ 4451.589440] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:17:38 (1788427058) [ 4457.785611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.209516] LustreError: 142391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.456256] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4473.474996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.974228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4473.981438] Lustre: Skipped 1 previous similar message [ 4480.074920] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4491.594765] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.850442] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.754623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.176757] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4519.412379] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.406848] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4528.684504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4528.996982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4533.694023] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4534.018622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4538.848726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.854735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4538.878751] Lustre: Skipped 2 previous similar messages [ 4538.902680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4538.919261] Lustre: Skipped 4 previous similar messages [ 4542.550967] Lustre: server umount lustre-OST0000 complete [ 4549.096409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.114338] Lustre: Skipped 2 previous similar messages [ 4558.304989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.315987] Lustre: Skipped 5 previous similar messages [ 4561.385839] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4561.591906] Lustre: server umount lustre-MDT0000 complete [ 4563.427433] LustreError: 142388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.449477] LustreError: 142388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4565.647135] LustreError: 142373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427173 with bad export cookie 15939525955540817356 [ 4565.660325] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4565.669671] LustreError: 142373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4565.928238] Lustre: server umount lustre-MDT0001 complete [ 4576.201040] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:19:43 (1788427183) [ 4584.544418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.774854] Lustre: 146596:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4584.806576] Lustre: 146596:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 4 previous similar messages [ 4600.351577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4605.535932] LustreError: 146583:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 4615.903518] Lustre: 146583:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4626.527786] Lustre: 146600:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4636.575275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.581243] Lustre: Skipped 1 previous similar message [ 4636.831392] LustreError: 146600:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4643.187578] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4651.780470] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4652.001428] Lustre: 146597:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4652.028846] Lustre: 146597:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 13 previous similar messages [ 4657.312967] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4659.447993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.229124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4665.781775] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4675.512219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.409440] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4690.055649] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4696.338345] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4700.790656] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4701.023387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4705.325873] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4705.471924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4711.909407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.912024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4711.921363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.931531] Lustre: Skipped 1 previous similar message [ 4715.533892] Lustre: server umount lustre-MDT0000 complete [ 4717.036637] LustreError: 146615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.062564] LustreError: 146615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4719.212513] LustreError: 146589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427327 with bad export cookie 15939525955540818511 [ 4719.218064] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4719.238354] LustreError: 146589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4719.740157] Lustre: server umount lustre-MDT0001 complete [ 4734.286891] Lustre: server umount lustre-OST0000 complete [ 4741.569464] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:22:29 (1788427349) [ 4748.079655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4763.681208] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4768.863532] LustreError: 150798:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 4779.440545] Lustre: 150798:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4784.095768] Lustre: 150812:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4784.123955] Lustre: 150812:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 8 previous similar messages [ 4789.919427] Lustre: 150815:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4800.159676] LustreError: 150815:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4807.222184] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4814.740268] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.639514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4821.317127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.999708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4826.179443] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4835.050990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.384297] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4848.412405] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4854.761672] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4864.428527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4864.595563] Lustre: lustre-OST0001: new disk, initializing [ 4864.598173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4864.603902] Lustre: 153678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4873.431803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4873.458259] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4873.672068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4874.370889] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4884.655856] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4889.431453] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4889.599490] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4894.247478] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4894.504701] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4897.249269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4897.270650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.291386] Lustre: Skipped 1 previous similar message [ 4897.302773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4897.314327] Lustre: Skipped 2 previous similar messages [ 4902.737520] Lustre: server umount lustre-OST0000 complete [ 4909.536126] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4909.544994] LustreError: Skipped 1 previous similar message [ 4913.896902] Lustre: server umount lustre-OST0001 complete [ 4927.459181] Lustre: server umount lustre-MDT0000 complete [ 4931.993966] LustreError: 150806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427539 with bad export cookie 15939525955540819596 [ 4932.006088] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4932.015379] LustreError: 150806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4932.479610] Lustre: server umount lustre-MDT0001 complete [ 4957.477571] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:26:04 (1788427564) [ 4959.097889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4960.540375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4962.093246] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:26:09 (1788427569) [ 4968.746680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.067842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4969.129322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4969.437890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.445726] Lustre: Skipped 6 previous similar messages [ 4973.919272] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4982.207485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.336663] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4982.368447] Lustre: 157241:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4986.634898] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4993.948625] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4999.455091] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5006.005882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.184692] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5012.455582] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5013.705980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5019.954406] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5024.617430] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5024.921155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5029.264695] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5029.489438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5034.468842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5034.475270] LustreError: Skipped 1 previous similar message [ 5034.481072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.496858] Lustre: Skipped 5 previous similar messages [ 5034.506870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5034.510023] Lustre: Skipped 8 previous similar messages [ 5037.445307] Lustre: server umount lustre-OST0000 complete [ 5058.556902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.272129] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5072.373622] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5074.234886] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.272213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.273281] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5074.302531] Lustre: Skipped 1 previous similar message [ 5079.067679] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5079.273932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5084.451960] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5084.712801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5090.807572] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d16c8f9e800.ost_server_uuid 50 [ 5092.309668] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d16c8f9e800.ost_server_uuid in FULL state after 0 sec [ 5092.486965] LustreError: 157248:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.60@tcp) returned error from blocking AST (req@00000000828e4128 x1875301262191232 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000046f8e48e/0xdd349267c421288e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.60@tcp remote: 0x90562d6c358d98ee expref: 5 pid: 158062 timeout: 5192 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5092.545216] LustreError: lustre-MDT0000: A client on nid 192.168.203.60@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5092.555458] LustreError: 157236:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.60@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000046f8e48e/0xdd349267c421288e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.60@tcp remote: 0x90562d6c358d98ee expref: 6 pid: 158062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5097.957641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5097.962206] LustreError: Skipped 1 previous similar message [ 5103.805418] Lustre: server umount lustre-OST0000 complete [ 5114.245774] Lustre: server umount lustre-MDT0000 complete [ 5115.364093] LustreError: 157248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.386573] LustreError: 157248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5118.256961] LustreError: 157231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427726 with bad export cookie 15939525955540820905 [ 5118.257511] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.267852] LustreError: 157231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5124.714592] Lustre: server umount lustre-MDT0001 complete [ 5136.200164] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5139.104244] Key type lgssc unregistered [ 5139.444639] LNet: 163355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5139.452142] LNetError: 163355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5139.492485] LNet: Removed LNI 192.168.203.160@tcp [ 5140.431657] Key type .llcrypt unregistered [ 5140.433430] Key type ._llcrypt unregistered [ 5164.851296] Key type ._llcrypt registered [ 5164.854695] Key type .llcrypt registered [ 5165.047959] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5165.961398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5166.001613] alg: No test for adler32 (adler32-zlib) [ 5167.305522] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5167.562806] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5169.226513] Key type lgssc registered [ 5170.590757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5179.473657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5179.506464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.051881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.780098] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5191.652346] LustreError: 164765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.677665] LustreError: 164765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5196.779048] LustreError: 164766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.898750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.694578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5203.336320] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5213.577389] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5221.143231] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5229.366985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5230.037722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.113245] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5245.293291] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5252.604355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5260.866950] Lustre: Failing over lustre-MDT0000 [ 5261.210071] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.241444] Lustre: server umount lustre-MDT0000 complete [ 5262.820267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.848284] Lustre: Skipped 2 previous similar messages [ 5267.939882] LustreError: 164762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.956044] LustreError: 164762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5268.965729] LustreError: mgc192.168.203.160@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 5268.969749] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5268.973968] LustreError: 164745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427876 with bad export cookie 7734133792173151662 [ 5268.986331] LustreError: Skipped 1 previous similar message [ 5269.017118] Lustre: 167483:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5269.020556] Lustre: 164757:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 5273.061197] Lustre: 167485:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5273.063501] LustreError: 164766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.100964] LustreError: 164766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5275.112390] Lustre: 167485:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5280.232267] Lustre: 167484:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5280.976842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.197960] Lustre: 164757:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5281.546955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.590157] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5286.202477] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5286.892217] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5286.909120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5288.629054] Lustre: lustre-MDT0000: Denying connection for new client ce26168a-57c9-4095-bb1b-e81f5d6ea600 (at 192.168.203.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5297.132093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.144988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.148188] Lustre: Skipped 1 previous similar message [ 5297.167695] Lustre: Skipped 1 previous similar message [ 5302.249595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5302.263347] Lustre: Skipped 1 previous similar message [ 5302.992190] Lustre: server umount lustre-OST0000 complete [ 5307.042440] LustreError: 168523:0:(ldlm_lib.c:3046:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5307.048851] Lustre: 167877:0:(ldlm_lib.c:2446:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5307.057307] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5307.079415] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5307.095358] LustreError: 167877:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cacb687b x1875302442652672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5307.119296] LustreError: 167877:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5307.136986] LustreError: 167877:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5307.151986] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5307.361386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.389743] LustreError: 164762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.405804] LustreError: 164762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5307.511149] Lustre: server umount lustre-MDT0000 complete [ 5311.679344] LustreError: 168235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427919 with bad export cookie 7734133792173152747 [ 5311.691182] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.696649] LustreError: 168235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5314.045773] Lustre: server umount lustre-MDT0001 complete [ 5326.177892] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5330.725904] Key type lgssc unregistered [ 5331.238515] LNet: 169276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5331.255107] LNetError: 169276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5331.277324] LNet: Removed LNI 192.168.203.160@tcp [ 5332.823434] Key type .llcrypt unregistered [ 5332.828279] Key type ._llcrypt unregistered [ 5360.106608] Key type ._llcrypt registered [ 5360.111457] Key type .llcrypt registered [ 5360.296542] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5361.774591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5361.864184] alg: No test for adler32 (adler32-zlib) [ 5363.025192] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5363.351183] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5365.071201] Key type lgssc registered [ 5366.505603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5375.450512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5375.504628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.043129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5382.392340] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5387.257926] LustreError: 170683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.269329] LustreError: 170683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5390.738196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.125274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5395.002728] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5401.373258] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5406.579779] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5412.508092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5412.762583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.723425] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5423.031691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5426.374382] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5432.294859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.307694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.324810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5433.321492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5433.325133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.336614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.445731] LustreError: 172506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.638841] Lustre: server umount lustre-OST0000 complete [ 5442.535952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5442.552144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.566474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.580463] Lustre: Skipped 1 previous similar message [ 5448.676365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.687694] Lustre: Skipped 3 previous similar messages [ 5448.789406] Lustre: server umount lustre-MDT0000 complete [ 5452.536570] LustreError: 170664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428060 with bad export cookie 3542061493952589569 [ 5452.543060] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.547019] LustreError: 170664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5452.970642] Lustre: server umount lustre-MDT0001 complete [ 5464.700196] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5467.641436] Key type lgssc unregistered [ 5467.986759] LNet: 174213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5468.006944] LNetError: 174213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5468.036234] LNet: Removed LNI 192.168.203.160@tcp [ 5469.034302] Key type .llcrypt unregistered [ 5469.037474] Key type ._llcrypt unregistered [ 5504.857371] Key type ._llcrypt registered [ 5504.860897] Key type .llcrypt registered [ 5505.089726] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5506.404816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5506.562115] alg: No test for adler32 (adler32-zlib) [ 5507.912695] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5508.296792] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5510.090244] Key type lgssc registered [ 5511.680435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5522.043917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5522.084123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.508710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.062297] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5533.676639] LustreError: 175915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.719422] LustreError: 175915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5538.552644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.785480] LustreError: 175914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.204888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.291508] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5552.040852] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.265673] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5566.069313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.473852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.533679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5572.621203] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5581.005297] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5594.081397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.097575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.127424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.105700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.115813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.116649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.159992] Lustre: Skipped 1 previous similar message [ 5598.745405] Lustre: server umount lustre-OST0000 complete [ 5605.344160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.347642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.372630] Lustre: Skipped 1 previous similar message [ 5605.399389] Lustre: Skipped 1 previous similar message [ 5610.466243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.474616] Lustre: Skipped 1 previous similar message [ 5615.600872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.609882] Lustre: Skipped 1 previous similar message [ 5617.119149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5617.482138] Lustre: server umount lustre-MDT0000 complete [ 5620.706026] LustreError: 175914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.731735] LustreError: 175914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5622.214206] LustreError: 175895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428230 with bad export cookie 9956128242383822020 [ 5622.217082] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5622.228719] LustreError: 175895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5622.800112] Lustre: server umount lustre-MDT0001 complete [ 5636.513865] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5640.671825] Key type lgssc unregistered [ 5641.375757] LNet: 179451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5641.412931] LNetError: 179451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5641.452812] LNet: Removed LNI 192.168.203.160@tcp [ 5643.188846] Key type .llcrypt unregistered [ 5643.190506] Key type ._llcrypt unregistered [ 5672.019984] Key type ._llcrypt registered [ 5672.025599] Key type .llcrypt registered [ 5672.164066] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5673.482443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.530251] alg: No test for adler32 (adler32-zlib) [ 5674.648659] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5674.919176] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5676.631995] Key type lgssc registered [ 5678.031492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5690.956358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5690.996987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.631246] Lustre: *** cfs_fail_loc=135, val=0*** [ 5692.725213] LustreError: 180974:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5692.730421] LustreError: 180974:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.203.160@tcp: cfg command failed: rc = -2 [ 5692.745257] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5692.745257] [ 5692.751447] LustreError: mgc192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5692.761386] LustreError: 180935:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5692.771486] LustreError: 180935:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 5692.777336] LustreError: 180935:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 5692.781517] LustreError: 180935:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 5692.889430] Lustre: server umount lustre-MDT0000 complete [ 5692.892677] LustreError: 180935:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5703.297737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.056989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5710.713885] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5714.403335] LustreError: 181346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.428937] LustreError: 181346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5719.536627] LustreError: 181345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.073871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.409339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5727.111331] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5736.447250] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5744.573762] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.180182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.202315] Lustre: Skipped 1 previous similar message [ 5747.203138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5747.236794] Lustre: Skipped 1 previous similar message [ 5752.290242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.296627] Lustre: Skipped 1 previous similar message [ 5753.582064] Lustre: server umount lustre-MDT0000 complete [ 5757.408847] LustreError: 181345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.423389] LustreError: 181345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5757.976471] LustreError: 181328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428365 with bad export cookie 3535273850251435223 [ 5757.982142] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.992389] LustreError: 181328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5758.700388] Lustre: server umount lustre-MDT0001 complete [ 5770.580929] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5773.180783] Key type lgssc unregistered [ 5773.604669] LNet: 183666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5773.613547] LNetError: 183666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5773.642761] LNet: Removed LNI 192.168.203.160@tcp [ 5774.797192] Key type .llcrypt unregistered [ 5774.802730] Key type ._llcrypt unregistered [ 5798.419996] Key type ._llcrypt registered [ 5798.424727] Key type .llcrypt registered [ 5798.511377] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5798.919376] Key type .llcrypt unregistered [ 5798.921985] Key type ._llcrypt unregistered [ 5817.229630] Key type ._llcrypt registered [ 5817.232327] Key type .llcrypt registered [ 5817.345731] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5819.186047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5819.259264] alg: No test for adler32 (adler32-zlib) [ 5820.586574] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5820.887581] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5822.559323] Key type lgssc registered [ 5823.549617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5833.187305] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5833.210949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.303568] Lustre: 185808:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5849.760721] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5854.879393] LustreError: 185795:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 5859.808526] Lustre: 185810:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5865.183588] Lustre: 185795:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 5875.496969] Lustre: 185808:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5876.575461] Lustre: 185812:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 5880.799883] Lustre: 185810:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5885.791229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5885.926497] Lustre: 185808:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5886.818987] LustreError: 185812:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 5894.584708] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5896.161318] Lustre: 185808:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5896.178467] Lustre: 185808:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 5902.817986] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5907.359672] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5910.809458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.204461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.260385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5916.648151] LustreError: 185824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.684573] LustreError: 185824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5917.061896] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5920.747413] LustreError: 185825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.804048] LustreError: 185825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5925.859022] LustreError: 186841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.958097] LustreError: 186841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5928.828361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.450134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5937.180961] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5949.045454] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5955.943694] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5959.316718] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5972.963274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5972.972496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.001279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.520759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.526456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.539454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.568557] Lustre: Skipped 1 previous similar message [ 5978.885705] Lustre: server umount lustre-OST0000 complete [ 5985.760450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.785424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.794257] Lustre: Skipped 1 previous similar message [ 5989.353541] Lustre: server umount lustre-MDT0000 complete [ 5990.880334] LustreError: 187272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.903056] LustreError: 187272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5993.076911] LustreError: 185802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428601 with bad export cookie 7492809583101696762 [ 5993.081297] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.101045] LustreError: 185802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5993.415248] Lustre: server umount lustre-MDT0001 complete [ 6005.312739] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6008.653443] Key type lgssc unregistered [ 6008.937856] LNet: 189819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6008.944796] LNetError: 189819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6008.965611] LNet: Removed LNI 192.168.203.160@tcp [ 6010.088351] Key type .llcrypt unregistered [ 6010.090790] Key type ._llcrypt unregistered [ 6037.039504] Key type ._llcrypt registered [ 6037.042382] Key type .llcrypt registered [ 6037.196245] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6038.400610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6038.565852] alg: No test for adler32 (adler32-zlib) [ 6039.917922] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6040.213643] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6041.919536] Key type lgssc registered [ 6043.997402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6053.526231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6053.547469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.935620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.904761] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6064.096432] LustreError: 191230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.109343] LustreError: 191230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6068.866150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.324809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6073.997908] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6081.631545] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6086.810842] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6092.923453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.334186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.483084] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6099.526460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6107.516492] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6112.757379] Lustre: Failing over lustre-MDT0000 [ 6113.096951] Lustre: server umount lustre-MDT0000 complete [ 6114.790920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.803767] LustreError: 191226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.814965] Lustre: Skipped 2 previous similar messages [ 6114.829526] LustreError: 191226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6119.906807] LustreError: 191231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.922523] LustreError: 191231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6120.959127] LustreError: mgc192.168.203.160@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 6120.966817] LustreError: 191210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428728 with bad export cookie 1309876439621177677 [ 6120.972572] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.976079] Lustre: 193980:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6120.977342] LustreError: Skipped 1 previous similar message [ 6120.978869] Lustre: 191222:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.203.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 6125.028548] Lustre: 193979:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6125.028858] LustreError: 191230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.091492] LustreError: 191230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6129.134113] Lustre: 193977:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6129.135077] LustreError: 193438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.176740] LustreError: 193438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6132.105456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.529200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6132.573743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6135.669490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6137.857883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6137.898905] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6137.967887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6138.525501] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6144.045188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6148.266749] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6159.843531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6159.854698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.875229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.869700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6160.893117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.920298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6161.647324] Lustre: server umount lustre-OST0000 complete [ 6168.550743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.560948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6168.581700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6172.760688] Lustre: server umount lustre-MDT0000 complete [ 6173.665234] LustreError: 191227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.689738] LustreError: 191227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6177.267381] LustreError: 191209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428785 with bad export cookie 1309876439621178776 [ 6177.275429] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.281298] LustreError: 191209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6177.858568] Lustre: server umount lustre-MDT0001 complete [ 6190.752562] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6194.620503] Key type lgssc unregistered [ 6195.042108] LNet: 196026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6195.052341] LNetError: 196026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6196.078883] LNet: Removed LNI 192.168.203.160@tcp [ 6197.400942] Key type .llcrypt unregistered [ 6197.407642] Key type ._llcrypt unregistered [ 6224.574607] Key type ._llcrypt registered [ 6224.580276] Key type .llcrypt registered [ 6224.711647] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6226.128872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6226.200536] alg: No test for adler32 (adler32-zlib) [ 6227.219601] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6227.502343] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6229.167270] Key type lgssc registered [ 6230.138712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.387972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.414159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.959232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.626807] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6248.421746] LustreError: 197433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.438398] LustreError: 197433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6251.362664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.640160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.697454] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6263.500538] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6269.206660] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6276.637873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6277.097470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.303566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6285.140825] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6293.358858] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6296.123347] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6302.435876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6320.413387] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6327.271421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.280491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.280561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.295610] Lustre: Skipped 1 previous similar message [ 6332.395626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.403692] Lustre: Skipped 2 previous similar messages [ 6333.257984] Lustre: server umount lustre-OST0000 complete [ 6337.510949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.532944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6337.550088] Lustre: Skipped 1 previous similar message [ 6342.545321] Lustre: server umount lustre-MDT0000 complete [ 6342.626195] LustreError: 197433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.641835] LustreError: 197433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6345.952161] LustreError: 198168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428953 with bad export cookie 6976520428064178945 [ 6345.962215] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.980927] LustreError: 198168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6346.368669] Lustre: server umount lustre-MDT0001 complete [ 6356.386974] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6359.398515] Key type lgssc unregistered [ 6359.694627] LNet: 201299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6359.701698] LNetError: 201299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6359.716567] LNet: Removed LNI 192.168.203.160@tcp [ 6360.603201] Key type .llcrypt unregistered [ 6360.607444] Key type ._llcrypt unregistered [ 6380.758655] Key type ._llcrypt registered [ 6380.763475] Key type .llcrypt registered [ 6380.938472] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6381.927353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6381.999654] alg: No test for adler32 (adler32-zlib) [ 6383.197203] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6383.410734] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6385.096067] Key type lgssc registered [ 6386.722936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6394.960405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6394.985971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.698596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.974157] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6406.115658] LustreError: 202562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.138222] LustreError: 202562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6411.235539] LustreError: 202563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.833358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.214151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6418.383357] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6429.382319] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6438.984679] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6448.092509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.542319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.147236] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6462.392956] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.989872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6471.700941] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6481.431322] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6489.882164] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6494.930502] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6510.051469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6510.064384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.103125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6513.125245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6513.139243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.162788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6515.569824] Lustre: server umount lustre-OST0000 complete [ 6520.803586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6520.813758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.823426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.828766] Lustre: Skipped 1 previous similar message [ 6523.363255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6523.371523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.482756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6528.496324] Lustre: Skipped 2 previous similar messages [ 6534.111228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6534.369417] Lustre: server umount lustre-MDT0000 complete [ 6538.338484] LustreError: 202544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429146 with bad export cookie 17679982466488360558 [ 6538.346159] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.363810] LustreError: 202544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6538.951033] Lustre: server umount lustre-MDT0001 complete [ 6550.235579] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6553.401118] Key type lgssc unregistered [ 6553.825332] LNet: 207488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6553.837461] LNetError: 207488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6553.869738] LNet: Removed LNI 192.168.203.160@tcp [ 6555.127232] Key type .llcrypt unregistered [ 6555.131960] Key type ._llcrypt unregistered [ 6585.839319] Key type ._llcrypt registered [ 6585.841526] Key type .llcrypt registered [ 6585.999838] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6588.089873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6588.107897] alg: No test for adler32 (adler32-zlib) [ 6589.211533] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6589.766480] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6591.495185] Key type lgssc registered [ 6593.172397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.605487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6604.661146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.169762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6611.938157] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6616.546083] LustreError: 208898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.575570] LustreError: 208898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6621.676975] LustreError: 208897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.019199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.417242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.595366] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6634.141350] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6639.830589] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.620943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6647.022676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.113766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6653.267489] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6661.879779] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.016749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6679.026415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.048796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.091615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.102190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.112652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.442727] Lustre: server umount lustre-OST0000 complete [ 6689.257049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6689.265144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.283919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6689.289633] Lustre: Skipped 1 previous similar message [ 6692.321334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6692.321343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.008879] Lustre: server umount lustre-MDT0000 complete [ 6697.447684] LustreError: 208897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.473672] LustreError: 208897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6698.926364] LustreError: 208879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429306 with bad export cookie 17252630977510604281 [ 6698.928654] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6698.937794] LustreError: 208879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6699.235626] Lustre: server umount lustre-MDT0001 complete [ 6708.771157] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6711.058718] Key type lgssc unregistered [ 6711.302955] LNet: 212583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6711.310091] LNetError: 212583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6711.330728] LNet: Removed LNI 192.168.203.160@tcp [ 6712.034389] Key type .llcrypt unregistered [ 6712.039270] Key type ._llcrypt unregistered [ 6729.407234] Key type ._llcrypt registered [ 6729.409231] Key type .llcrypt registered [ 6729.550597] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6730.484926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6730.507117] alg: No test for adler32 (adler32-zlib) [ 6731.537681] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6731.863170] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6733.519180] Key type lgssc registered [ 6734.623791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6742.111292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6742.125252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.668896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.570923] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6753.255058] LustreError: 213845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.278333] LustreError: 213845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6756.647810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.077745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6761.372120] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6768.831563] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6774.855866] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6782.395285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6783.013242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.146537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6792.622995] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6804.598657] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6821.860401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.888911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.331501] Lustre: server umount lustre-OST0000 complete [ 6832.100957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6832.102850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6832.113227] Lustre: Skipped 2 previous similar messages [ 6832.123689] Lustre: Skipped 2 previous similar messages [ 6837.216987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6837.225695] Lustre: Skipped 1 previous similar message [ 6841.314459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.359196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6843.587808] Lustre: server umount lustre-MDT0000 complete [ 6846.436572] LustreError: 215857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.450162] LustreError: 215857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6847.592980] LustreError: 213823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429455 with bad export cookie 7795883308463967101 [ 6847.610789] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6847.611994] LustreError: 213823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6848.025267] Lustre: server umount lustre-MDT0001 complete [ 6857.763360] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6860.449603] Key type lgssc unregistered [ 6860.789579] LNet: 217528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6860.796278] LNetError: 217528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6860.832585] LNet: Removed LNI 192.168.203.160@tcp [ 6861.721663] Key type .llcrypt unregistered [ 6861.726632] Key type ._llcrypt unregistered [ 6879.259695] Key type ._llcrypt registered [ 6879.261449] Key type .llcrypt registered [ 6879.371581] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6880.435382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6880.474847] alg: No test for adler32 (adler32-zlib) [ 6881.717361] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6882.000882] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6883.695163] Key type lgssc registered [ 6885.281413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6895.777193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6895.802963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.527129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6904.515594] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6907.878740] LustreError: 218789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.919206] LustreError: 218789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6912.995296] LustreError: 218790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.126992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.531122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.693061] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6930.612593] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6935.983422] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6942.496235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.911676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.774951] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6956.367212] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6967.491971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6977.015786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.127540] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6977.504940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6984.319743] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6991.760927] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6993.576721] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7006.253627] Lustre: Permanently deactivating lustre-OST0001 [ 7006.292973] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7016.347891] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7026.245061] Lustre: Permanently reactivating lustre-OST0001 [ 7026.281061] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7026.296341] Lustre: Skipped 2 previous similar messages [ 7032.181369] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.191386] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7032.201116] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7032.211112] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7032.251254] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7037.413656] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.429053] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.456472] Lustre: Skipped 1 previous similar message [ 7037.459256] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7037.469828] Lustre: Skipped 1 previous similar message [ 7038.374959] Lustre: server umount lustre-OST0001 complete [ 7041.003284] LustreError: 220612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.046923] LustreError: 220612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7043.552265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.572404] LustreError: Skipped 1 previous similar message [ 7043.584041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.607955] Lustre: Skipped 1 previous similar message [ 7043.620514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7044.578967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.120373] LustreError: 220614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.119863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.153747] Lustre: Skipped 2 previous similar messages [ 7048.519261] Lustre: server umount lustre-OST0000 complete [ 7054.816611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7054.840042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7054.862110] Lustre: Skipped 1 previous similar message [ 7054.875704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.480123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.489821] Lustre: Skipped 3 previous similar messages [ 7067.103133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7067.335747] Lustre: server umount lustre-MDT0000 complete [ 7070.703698] LustreError: 218770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429678 with bad export cookie 10184995427551325206 [ 7070.709607] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7070.726368] LustreError: 218770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7070.998571] Lustre: server umount lustre-MDT0001 complete [ 7080.773724] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7083.394540] Key type lgssc unregistered [ 7083.692357] LNet: 223923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7083.703383] LNetError: 223923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7083.729218] LNet: Removed LNI 192.168.203.160@tcp [ 7084.532288] Key type .llcrypt unregistered [ 7084.535565] Key type ._llcrypt unregistered [ 7121.353232] Key type ._llcrypt registered [ 7121.358484] Key type .llcrypt registered [ 7121.464692] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7122.630230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.778686] alg: No test for adler32 (adler32-zlib) [ 7123.795126] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7123.989539] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7125.655163] Key type lgssc registered [ 7126.749410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7134.272517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7134.308526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.641657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7135.696910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7135.934123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7140.943556] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7149.499148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.619521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7149.655629] Lustre: 226125:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7149.984297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7154.904854] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7162.727737] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.473239] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7174.912721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.122725] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7175.342439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.553702] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7185.101384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7189.291156] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7193.501550] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7198.210338] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7206.257918] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7213.986514] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7232.265750] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7232.276747] Lustre: Skipped 1 previous similar message [ 7253.217337] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7253.222732] Lustre: Skipped 2 previous similar messages [ 7287.991284] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7288.000747] Lustre: Skipped 3 previous similar messages [ 7353.574921] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7353.580456] Lustre: Skipped 7 previous similar messages [ 7369.698101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.706901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7369.708510] Lustre: Skipped 1 previous similar message [ 7369.725756] Lustre: Skipped 1 previous similar message [ 7370.539121] Lustre: server umount lustre-OST0000 complete [ 7374.822994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.829106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.839330] Lustre: Skipped 1 previous similar message [ 7379.939529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7379.951554] Lustre: Skipped 2 previous similar messages [ 7380.583370] Lustre: server umount lustre-MDT0000 complete [ 7383.661078] LustreError: 226117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429991 with bad export cookie 6977523755878515208 [ 7383.668524] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7384.028761] Lustre: server umount lustre-MDT0001 complete [ 7394.003769] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7396.959729] Key type lgssc unregistered [ 7397.165443] LNet: 230921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7397.173180] LNetError: 230921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7397.189232] LNet: Removed LNI 192.168.203.160@tcp [ 7398.027856] Key type .llcrypt unregistered [ 7398.034717] Key type ._llcrypt unregistered [ 7423.028909] Key type ._llcrypt registered [ 7423.031641] Key type .llcrypt registered [ 7423.152351] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7424.243824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7424.391552] alg: No test for adler32 (adler32-zlib) [ 7425.442369] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7425.768089] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7427.447514] Key type lgssc registered [ 7428.537701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7436.420407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7436.448197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.045179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.865713] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7448.545401] LustreError: 232329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.576513] LustreError: 232329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7452.050533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.345669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7457.150490] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7464.725532] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7470.357132] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.207383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.576434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.715674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7484.132658] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 7492.234496] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.547759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7512.565737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.582781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.607705] Lustre: Skipped 1 previous similar message [ 7512.639389] Lustre: Skipped 1 previous similar message [ 7517.664509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7517.676637] Lustre: Skipped 1 previous similar message [ 7522.786799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7522.812174] Lustre: Skipped 1 previous similar message [ 7526.367138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.559474] Lustre: server umount lustre-OST0000 complete [ 7533.025095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.028434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.056969] Lustre: Skipped 1 previous similar message [ 7533.069707] Lustre: Skipped 1 previous similar message [ 7538.149691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7538.156358] Lustre: Skipped 1 previous similar message [ 7544.288674] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7544.435474] Lustre: server umount lustre-MDT0000 complete [ 7547.437307] LustreError: 234911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430155 with bad export cookie 15557893707138994071 [ 7547.447643] LustreError: 234911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7547.453180] LustreError: mgc192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7547.704179] Lustre: server umount lustre-MDT0001 complete [ 7556.686121] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7559.205440] Key type lgssc unregistered [ 7559.518761] LNet: 236024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.525646] LNetError: 236024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7559.546081] LNet: Removed LNI 192.168.203.160@tcp [ 7560.310585] Key type .llcrypt unregistered [ 7560.313421] Key type ._llcrypt unregistered [ 7581.388357] Key type ._llcrypt registered [ 7581.399383] Key type .llcrypt registered [ 7581.638798] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7582.701071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7582.720498] alg: No test for adler32 (adler32-zlib) [ 7583.743674] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7583.975157] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7585.657920] Key type lgssc registered [ 7587.035027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7857.769855] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7862.076400] Key type lgssc unregistered [ 7862.560156] LNet: 238037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.568767] LNetError: 238037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.625025] LNet: Removed LNI 192.168.203.160@tcp [ 7863.830848] Key type .llcrypt unregistered [ 7863.834767] Key type ._llcrypt unregistered [ 7893.263368] Key type ._llcrypt registered [ 7893.265881] Key type .llcrypt registered [ 7893.424914] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 7909.724692] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 7911.117567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7911.159351] alg: No test for adler32 (adler32-zlib) [ 7912.322894] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7912.572596] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 7914.255157] Key type lgssc registered [ 7915.506766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7945.725928] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7657 sec ========== 06:15:53 (1788430553) [ 7947.349660] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:15:54 (1788430554) === [ 7950.560612] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:15:57 (1788430557) === [ 7988.800818] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 7991.822969] Key type lgssc unregistered [ 7992.176498] LNet: 243269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7992.192640] LNetError: 243269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7992.236020] LNet: Removed LNI 192.168.203.160@tcp [ 7993.229460] Key type .llcrypt unregistered [ 7993.233100] Key type ._llcrypt unregistered