[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476908026 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003342] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.007544] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.022586] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027715] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029139] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030131] *** VALIDATE tmpfs *** [ 0.032459] *** VALIDATE proc *** [ 0.034125] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039068] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.045020] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.048330] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050125] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052009] ... generic registers: 4 [ 0.053011] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055013] ... fixed-purpose events: 3 [ 0.056012] ... event mask: 000000070000000f [ 0.057291] rcu: Hierarchical SRCU implementation. [ 0.059742] smp: Bringing up secondary CPUs ... [ 0.060747] x86: Booting SMP configuration: [ 0.061026] .... node #0, CPUs: #1 #2 #3 [ 0.071192] smp: Brought up 1 node, 4 CPUs [ 0.073013] smpboot: Max logical packages: 1 [ 0.074019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106566] node 0 deferred pages initialised in 30ms [ 0.110136] devtmpfs: initialized [ 0.111379] x86/mm: Memory block size: 128MB [ 0.115108] gcov: version magic: 0x41383552 [ 0.118326] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121273] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123297] pinctrl core: initialized pinctrl subsystem [ 0.125191] [ 0.125655] ************************************************************* [ 0.127011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129011] ** ** [ 0.130010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.132014] ** ** [ 0.134009] ** This means that this kernel is built to expose internal ** [ 0.136009] ** IOMMU data structures, which may compromise security on ** [ 0.137007] ** your system. ** [ 0.139008] ** ** [ 0.140007] ** If you see this message and you are not debugging the ** [ 0.143012] ** kernel, report this immediately to your vendor! ** [ 0.145013] ** ** [ 0.147010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149011] ************************************************************* [ 0.151774] NET: Registered protocol family 16 [ 0.154434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162608] cpuidle: using governor menu [ 0.164694] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167510] PCI: Using configuration type 1 for base access [ 0.169123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178039] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181060] cryptd: max_cpu_qlen set to 1000 [ 0.182203] ACPI: Added _OSI(Module Device) [ 0.183009] ACPI: Added _OSI(Processor Device) [ 0.183910] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.185021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190483] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.196378] ACPI: Interpreter enabled [ 0.197045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.198007] ACPI: Using IOAPIC for interrupt routing [ 0.199071] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.209763] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219037] acpiphp: Slot [2] registered [ 0.220009] acpiphp: Slot [5] registered [ 0.220788] acpiphp: Slot [6] registered [ 0.222071] acpiphp: Slot [7] registered [ 0.223132] acpiphp: Slot [8] registered [ 0.224090] acpiphp: Slot [9] registered [ 0.225093] acpiphp: Slot [10] registered [ 0.226131] acpiphp: Slot [3] registered [ 0.228114] acpiphp: Slot [4] registered [ 0.229101] acpiphp: Slot [11] registered [ 0.231133] acpiphp: Slot [12] registered [ 0.233112] acpiphp: Slot [13] registered [ 0.234108] acpiphp: Slot [14] registered [ 0.235074] acpiphp: Slot [15] registered [ 0.236065] acpiphp: Slot [16] registered [ 0.237130] acpiphp: Slot [17] registered [ 0.238062] acpiphp: Slot [18] registered [ 0.239061] acpiphp: Slot [19] registered [ 0.239940] acpiphp: Slot [20] registered [ 0.241062] acpiphp: Slot [21] registered [ 0.242016] acpiphp: Slot [22] registered [ 0.243059] acpiphp: Slot [23] registered [ 0.244060] acpiphp: Slot [24] registered [ 0.246113] acpiphp: Slot [25] registered [ 0.247121] acpiphp: Slot [26] registered [ 0.249131] acpiphp: Slot [27] registered [ 0.250133] acpiphp: Slot [28] registered [ 0.252133] acpiphp: Slot [29] registered [ 0.253092] acpiphp: Slot [30] registered [ 0.254067] acpiphp: Slot [31] registered [ 0.255016] PCI host bridge to bus 0000:00 [ 0.255899] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.257017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.259016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.264018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.265156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266762] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268917] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275630] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278755] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.280018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.282015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.283014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.286331] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.287585] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.289033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.291878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.297019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306749] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310666] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317011] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.322013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.327013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.352917] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.356018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.361016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.375022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.382928] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.388016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.392014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.404014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.411377] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.415015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.420017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.430026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.437054] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.442014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.446015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.458019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.465376] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.470018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.476011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.486019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.496275] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.499297] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.501434] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.504411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.507330] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.512599] iommu: Default domain type: Passthrough [ 0.514650] SCSI subsystem initialized [ 0.517234] ACPI: bus type USB registered [ 0.518140] usbcore: registered new interface driver usbfs [ 0.521330] usbcore: registered new interface driver hub [ 0.523187] usbcore: registered new device driver usb [ 0.525242] pps_core: LinuxPPS API ver. 1 registered [ 0.527020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.530091] PTP clock support registered [ 0.533118] EDAC MC: Ver: 3.0.0 [ 0.535189] PCI: Using ACPI for IRQ routing [ 0.537917] NetLabel: Initializing [ 0.539014] NetLabel: domain hash size = 128 [ 0.540011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.542102] NetLabel: unlabeled traffic allowed by default [ 0.544127] vgaarb: loaded [ 0.545255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.547011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.553288] clocksource: Switched to clocksource kvm-clock [ 0.664090] VFS: Disk quotas dquot_6.6.0 [ 0.665811] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.668550] *** VALIDATE ramfs *** [ 0.669838] *** VALIDATE hugetlbfs *** [ 0.671606] pnp: PnP ACPI init [ 0.673976] pnp: PnP ACPI: found 6 devices [ 0.702914] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.706164] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.708388] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.710042] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.712629] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.714388] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.716562] NET: Registered protocol family 2 [ 0.718475] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.722849] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.725706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.731122] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.734735] TCP: Hash tables configured (established 65536 bind 65536) [ 0.737762] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.741090] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.744183] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747450] NET: Registered protocol family 1 [ 0.750720] RPC: Registered named UNIX socket transport module. [ 0.753240] RPC: Registered udp transport module. [ 0.754849] RPC: Registered tcp transport module. [ 0.756581] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.759255] NET: Registered protocol family 44 [ 0.761199] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.762747] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.764260] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.766052] PCI: CLS 0 bytes, default 64 [ 0.766946] Unpacking initramfs... [ 2.175905] debug: unmapping init [mem 0xffff8d647cc54000-0xffff8d647ffbffff] [ 2.179779] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.182056] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.183932] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.663729] Initialise system trusted keyrings [ 2.665427] Key type blacklist registered [ 2.667158] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.675658] zbud: loaded [ 2.678524] *** VALIDATE nfs *** [ 2.679591] *** VALIDATE nfs4 *** [ 2.680753] pstore: using deflate compression [ 2.683807] Platform Keyring initialized [ 2.780943] NET: Registered protocol family 38 [ 2.782236] Key type asymmetric registered [ 2.783197] Asymmetric key parser 'x509' registered [ 2.784588] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.786926] io scheduler mq-deadline registered [ 2.788599] io scheduler kyber registered [ 2.789952] io scheduler bfq registered [ 2.792370] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.794519] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.796359] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.798179] ACPI: Power Button [PWRF] [ 2.802044] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.807550] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.814785] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.818851] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.827219] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.852228] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.879578] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.884365] Non-volatile memory driver v1.3 [ 2.885991] Linux agpgart interface v0.103 [ 2.915062] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 2.917373] vda: detected capacity change from 0 to 34836480 [ 2.928917] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.930903] vdb: detected capacity change from 0 to 1073741824 [ 2.942886] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.945303] vdc: detected capacity change from 0 to 2621440000 [ 2.960460] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.963154] vdd: detected capacity change from 0 to 2621440000 [ 2.976072] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.978270] vde: detected capacity change from 0 to 4294967296 [ 2.992022] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.994912] vdf: detected capacity change from 0 to 4294967296 [ 3.004241] libphy: Fixed MDIO Bus: probed [ 3.013568] usbcore: registered new interface driver usbserial_generic [ 3.015563] usbserial: USB Serial support registered for generic [ 3.017727] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.021667] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.023464] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.025635] mousedev: PS/2 mouse device common for all mice [ 3.028483] rtc_cmos 00:05: RTC can wake from S4 [ 3.031905] rtc_cmos 00:05: registered as rtc0 [ 3.032062] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.034156] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.040227] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.040532] intel_pstate: CPU model not supported [ 3.045322] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.047431] hid: raw HID events driver (C) Jiri Kosina [ 3.047594] usbcore: registered new interface driver usbhid [ 3.050832] usbhid: USB HID core driver [ 3.051867] drop_monitor: Initializing network drop monitor service [ 3.054050] Initializing XFRM netlink socket [ 3.056082] NET: Registered protocol family 10 [ 3.059580] Segment Routing with IPv6 [ 3.061415] NET: Registered protocol family 17 [ 3.063559] mpls_gso: MPLS GSO support [ 3.069425] RAS: Correctable Errors collector initialized. [ 3.071475] AVX version of gcm_enc/dec engaged. [ 3.073090] AES CTR mode by8 optimization enabled [ 3.142889] sched_clock: Marking stable (3142866296, 0)->(4077507397, -934641101) [ 3.145683] registered taskstats version 1 [ 3.147615] Loading compiled-in X.509 certificates [ 3.148915] zswap: loaded using pool lzo/zbud [ 3.169677] Key type big_key registered [ 3.179442] Key type encrypted registered [ 3.180874] ima: No TPM chip found, activating TPM-bypass! [ 3.182576] ima: Allocated hash algorithm: sha1 [ 3.184243] ima: No architecture policies found [ 3.185418] evm: Initialising EVM extended attributes: [ 3.186507] evm: security.selinux [ 3.187376] evm: security.ima [ 3.188033] evm: security.capability [ 3.188893] evm: HMAC attrs: 0x1 [ 3.190592] rtc_cmos 00:05: setting system clock to 2026-04-14 17:24:08 UTC (1776187448) [ 3.195233] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.197543] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.205107] Write protecting the kernel read-only data: 28672k [ 3.208466] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.210724] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.239682] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.248171] systemd[1]: Detected virtualization kvm. [ 3.250094] systemd[1]: Detected architecture x86-64. [ 3.252066] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.279282] systemd[1]: No hostname configured. [ 3.280543] systemd[1]: Set hostname to . [ 3.281867] random: systemd: uninitialized urandom read (16 bytes read) [ 3.283710] systemd[1]: Initializing machine ID from random generator. [ 3.390343] random: systemd: uninitialized urandom read (16 bytes read) [ 3.392399] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.396096] random: systemd: uninitialized urandom read (16 bytes read) [ 3.398498] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.401895] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.885692] device-mapper: uevent: version 1.0.3 [ 3.887060] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.484494] virtio_net virtio0 ens2: renamed from eth0 [ 4.491560] random: fast init done [ 4.556157] scsi host0: ata_piix [ 4.571700] scsi host1: ata_piix [ 4.573295] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.575089] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.136206] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.696924] random: crng init done [ 9.697903] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.850048] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.814638] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.041914] SELinux: Disabled at runtime. [ 11.104161] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.109748] systemd[1]: Detected virtualization kvm. [ 11.111017] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.510260] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.513812] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.521420] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.523951] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.525974] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.532965] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.539946] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ 11.627109] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 11.846565] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.069894] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.078278] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.199261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.220466] EDAC sbridge: Ver: 1.1.2 [ 13.607421] Key type dns_resolver registered [ 13.898666] NFS: Registering the id_resolver key type [ 13.900632] Key type id_resolver registered [ 13.902302] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-server login: [ 63.215015] hrtimer: interrupt took 5006195 ns [ 91.663494] libcfs: loading out-of-tree module taints kernel. [ 91.714478] alg: No test for adler32 (adler32-zlib) [ 92.469488] Key type ._llcrypt registered [ 92.472278] Key type .llcrypt registered [ 92.579622] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 114.127845] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 116.040792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.975443] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 117.848947] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 117.852530] LNet: Accept secure, port 988 [ 119.672840] Key type lgssc registered [ 121.733914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.174373] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.391506] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.171659] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 162.732890] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.976132] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.925919] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 196.380447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.446458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.679782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 197.706414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 197.775822] Lustre: lustre-MDT0000: new disk, initializing [ 197.831721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.839502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.720581] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 215.198743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.266227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.305481] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 215.325437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 215.329938] Lustre: Skipped 1 previous similar message [ 215.385313] Lustre: lustre-MDT0001: new disk, initializing [ 215.424427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.440046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.448248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.454761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 230.453607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 230.522097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.526383] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 230.699891] Lustre: lustre-OST0000: new disk, initializing [ 230.703688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 230.755787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.705872] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 238.612255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.628369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.656397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 247.814941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.832413] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 248.026880] Lustre: lustre-OST0001: new disk, initializing [ 248.031095] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 248.138899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.172413] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 256.529471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.537347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.773629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.936540] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 281.462957] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.607270] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing yml_node [ 289.856419] Lustre: DEBUG MARKER: Client: 2.15.8.12 [ 291.989863] Lustre: DEBUG MARKER: MDS: 2.15.8.12 [ 294.128490] Lustre: DEBUG MARKER: OSS: 2.15.8.12 [ 295.869841] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 13:28:59 EDT 2026 [ 304.129834] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 305.537810] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 323.048542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.052077] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.053188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.059073] Lustre: Skipped 1 previous similar message [ 327.456792] Lustre: server umount lustre-MDT0000 complete [ 328.162375] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.179923] LustreError: Skipped 3 previous similar messages [ 330.599871] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187775 with bad export cookie 8359026684401325965 [ 330.601276] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.606778] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.287544] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.292568] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.301366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.301376] Lustre: Skipped 3 previous similar messages [ 333.306711] Lustre: Skipped 2 previous similar messages [ 333.344159] LustreError: Skipped 1 previous similar message [ 335.328955] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.329750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.364485] Lustre: Skipped 2 previous similar messages [ 336.917917] Lustre: server umount lustre-MDT0001 complete [ 340.470361] Lustre: server umount lustre-OST0000 complete [ 344.077323] Lustre: server umount lustre-OST0001 complete [ 350.169692] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 356.920816] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 365.019295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 371.572559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 378.383885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 385.742536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.281070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.395561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.694378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 393.722830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 393.806339] Lustre: lustre-MDT0000: new disk, initializing [ 393.892713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.912438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.244939] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 407.743437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.829338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.933369] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 407.989293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 407.994136] Lustre: Skipped 1 previous similar message [ 408.089957] Lustre: lustre-MDT0001: new disk, initializing [ 408.161124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.226532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.252978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.273816] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 425.237704] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 431.379175] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 437.675293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 437.742347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.747176] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 437.953868] Lustre: lustre-OST0000: new disk, initializing [ 437.958322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 438.045358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.140510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.146492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.307825] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 452.368470] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 456.491427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 456.647873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.499325] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 460.660966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 464.867231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.870082] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.871338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.871345] Lustre: Skipped 2 previous similar messages [ 464.878065] Lustre: Skipped 1 previous similar message [ 464.882324] LustreError: Skipped 1 previous similar message [ 468.460760] Lustre: server umount lustre-OST0000 complete [ 474.593749] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.598597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.607791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.617612] Lustre: Skipped 1 previous similar message [ 486.368589] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 486.381356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.406344] Lustre: Skipped 7 previous similar messages [ 486.484902] Lustre: server umount lustre-MDT0000 complete [ 489.912641] LustreError: 14336:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187935 with bad export cookie 8359026684401327421 [ 489.928843] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.259677] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:32:17 (1776187937) [ 500.172419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.648758] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.664369] LustreError: Skipped 3 previous similar messages [ 500.719393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.254296] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 505.825660] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.266286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.262695] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 524.608904] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 530.189404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 535.942297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.002968] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 547.219575] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 562.153196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.153915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.166484] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 562.168837] Lustre: Skipped 2 previous similar messages [ 562.179557] Lustre: Skipped 1 previous similar message [ 566.059313] Lustre: server umount lustre-OST0000 complete [ 566.062114] Lustre: Skipped 1 previous similar message [ 569.831920] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 572.386418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.398477] Lustre: Skipped 1 previous similar message [ 577.509691] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.523845] LustreError: Skipped 2 previous similar messages [ 579.186885] LustreError: 18732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188024 with bad export cookie 8359026684401328275 [ 579.188116] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.203872] LustreError: 18732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 590.535872] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 593.008828] Key type lgssc unregistered [ 593.255885] LNet: 22427:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.269850] LNet: Removed LNI 192.168.202.103@tcp [ 593.965518] Key type .llcrypt unregistered [ 593.967706] Key type ._llcrypt unregistered [ 613.619582] alg: No test for adler32 (adler32-zlib) [ 614.380586] Key type ._llcrypt registered [ 614.384144] Key type .llcrypt registered [ 614.595907] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 615.967567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.611995] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 616.898657] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 616.902839] LNet: Accept secure, port 988 [ 618.592132] Key type lgssc registered [ 619.572332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.158801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.685533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.754959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.850431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.501373] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 639.976719] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.047105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.302484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 648.952430] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 658.220720] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 665.723347] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 671.521870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.957916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.735691] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 681.210863] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 685.013855] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.947384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.948584] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 701.967497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 706.419126] Lustre: server umount lustre-OST0000 complete [ 710.125722] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 710.141136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.155130] Lustre: Skipped 1 previous similar message [ 710.160131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.162979] Lustre: Skipped 1 previous similar message [ 712.164162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.165469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.300972] Lustre: server umount lustre-MDT0000 complete [ 717.290586] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.386547] LustreError: 23871:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188163 with bad export cookie 15545837265063817151 [ 718.394110] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.415140] LustreError: 23871:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 718.642090] Lustre: server umount lustre-MDT0001 complete [ 729.314111] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 731.815643] Key type lgssc unregistered [ 732.049269] LNet: 27891:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.062894] LNet: Removed LNI 192.168.202.103@tcp [ 732.846627] Key type .llcrypt unregistered [ 732.851266] Key type ._llcrypt unregistered [ 750.713428] alg: No test for adler32 (adler32-zlib) [ 751.465412] Key type ._llcrypt registered [ 751.467322] Key type .llcrypt registered [ 751.570480] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 752.528180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 752.879259] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 753.042857] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 753.047390] LNet: Accept secure, port 988 [ 754.664346] Key type lgssc registered [ 755.534063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 762.539598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.001603] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.082314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.706802] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 769.509249] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 774.629102] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.437263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.865080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 781.392401] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 787.666794] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 792.407848] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 804.496519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.762152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.833718] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 813.998831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 815.699770] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 821.737538] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 821.742955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.760292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 823.265837] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.274246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.285669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 824.307875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 826.741238] Lustre: server umount lustre-OST0000 complete [ 834.021135] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 834.031909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.041954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 834.055206] Lustre: Skipped 1 previous similar message [ 836.864957] Lustre: server umount lustre-MDT0000 complete [ 839.649622] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 839.660673] LustreError: Skipped 1 previous similar message [ 840.111151] LustreError: 32343:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188285 with bad export cookie 7349539357533545746 [ 840.116466] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 840.125037] LustreError: 32343:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 840.433524] Lustre: server umount lustre-MDT0001 complete [ 850.923570] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 853.764983] Key type lgssc unregistered [ 854.239523] LNet: 33348:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.260495] LNet: Removed LNI 192.168.202.103@tcp [ 855.271805] Key type .llcrypt unregistered [ 855.274370] Key type ._llcrypt unregistered [ 875.838382] alg: No test for adler32 (adler32-zlib) [ 876.602259] Key type ._llcrypt registered [ 876.604692] Key type .llcrypt registered [ 876.737963] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 877.474260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.227709] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 878.467339] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 878.473067] LNet: Accept secure, port 988 [ 880.129092] Key type lgssc registered [ 881.229030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 888.823199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.082450] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.118245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.716338] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 895.462244] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.578543] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.672163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.938773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.304433] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 914.956248] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.376711] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 926.267552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.533390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.843563] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 937.683884] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 939.969969] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 953.316293] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.329431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.359037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.363166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.365228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.486705] Lustre: server umount lustre-OST0000 complete [ 964.577375] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.580990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.597574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.600549] Lustre: Skipped 1 previous similar message [ 968.424431] Lustre: server umount lustre-MDT0000 complete [ 970.722523] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.747220] LustreError: Skipped 1 previous similar message [ 972.578924] LustreError: 34790:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188417 with bad export cookie 5521482566426346264 [ 972.581685] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.596449] LustreError: 34790:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 973.007704] Lustre: server umount lustre-MDT0001 complete [ 987.499834] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 991.562305] Key type lgssc unregistered [ 992.117916] LNet: 38494:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.132166] LNet: Removed LNI 192.168.202.103@tcp [ 992.902230] Key type .llcrypt unregistered [ 992.904438] Key type ._llcrypt unregistered [ 1018.438934] alg: No test for adler32 (adler32-zlib) [ 1019.204152] Key type ._llcrypt registered [ 1019.210781] Key type .llcrypt registered [ 1019.439269] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1020.951650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.200912] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1021.367584] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1021.377226] LNet: Accept secure, port 988 [ 1023.120429] Key type lgssc registered [ 1024.751540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.403433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.902740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.945976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.581541] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1040.355300] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.476714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.757920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.482873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1054.914519] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1064.275833] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.627465] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1079.196736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.521493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.134151] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1086.706768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1090.333204] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1096.165045] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.177051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.198424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.187345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.190535] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1097.191569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.330532] Lustre: server umount lustre-OST0000 complete [ 1122.274044] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1122.277153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.289271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.292761] Lustre: Skipped 2 previous similar messages [ 1127.975594] Lustre: server umount lustre-MDT0000 complete [ 1128.935933] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.959261] LustreError: Skipped 1 previous similar message [ 1131.197226] LustreError: 39939:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188576 with bad export cookie 17439777128788940796 [ 1131.201487] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.206642] LustreError: 39939:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1137.635775] Lustre: server umount lustre-MDT0001 complete [ 1150.853598] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1154.057304] Key type lgssc unregistered [ 1154.423733] LNet: 43786:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.429721] LNet: Removed LNI 192.168.202.103@tcp [ 1155.227590] Key type .llcrypt unregistered [ 1155.229627] Key type ._llcrypt unregistered [ 1178.108795] alg: No test for adler32 (adler32-zlib) [ 1178.867942] Key type ._llcrypt registered [ 1178.871652] Key type .llcrypt registered [ 1179.049934] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1180.781851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.128528] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1181.360477] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1181.364862] LNet: Accept secure, port 988 [ 1183.088360] Key type lgssc registered [ 1184.459643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.387568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.796926] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.835229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.939344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1201.122152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.244217] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.343306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.675240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1215.912716] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1224.482619] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.124113] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1237.296933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.872412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.417466] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1246.061907] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1250.672924] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1257.530769] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1261.550730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.555561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.573789] Lustre: Skipped 2 previous similar messages [ 1262.607432] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (stopping) [ 1262.625620] Lustre: Skipped 2 previous similar messages [ 1263.636681] Lustre: server umount lustre-MDT0000 complete [ 1266.658419] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.674043] LustreError: Skipped 2 previous similar messages [ 1267.744808] LustreError: 45232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188713 with bad export cookie 12806307651553828877 [ 1267.748093] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.761162] LustreError: 45232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1271.777667] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.778600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.798137] LustreError: Skipped 1 previous similar message [ 1271.845180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1274.154883] Lustre: server umount lustre-MDT0001 complete [ 1289.246688] Lustre: server umount lustre-OST0000 complete [ 1308.643855] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1312.191689] Key type lgssc unregistered [ 1312.453603] LNet: 49232:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.464805] LNet: Removed LNI 192.168.202.103@tcp [ 1313.714604] Key type .llcrypt unregistered [ 1313.717407] Key type ._llcrypt unregistered [ 1339.198791] alg: No test for adler32 (adler32-zlib) [ 1339.955917] Key type ._llcrypt registered [ 1339.958978] Key type .llcrypt registered [ 1340.089991] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1341.046303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1341.908784] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1342.114345] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1342.126528] LNet: Accept secure, port 988 [ 1343.832161] Key type lgssc registered [ 1344.980507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.038163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.720434] LustreError: 50670:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 1401.440289] LustreError: 50670:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 1431.200959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.621966] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1442.991928] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.063746] Lustre: server umount lustre-OST0000 complete [ 1502.336464] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1505.494634] Key type lgssc unregistered [ 1505.751829] LNet: 52342:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.765343] LNet: Removed LNI 192.168.202.103@tcp [ 1506.462667] Key type .llcrypt unregistered [ 1506.466231] Key type ._llcrypt unregistered [ 1531.420931] alg: No test for adler32 (adler32-zlib) [ 1532.176287] Key type ._llcrypt registered [ 1532.178809] Key type .llcrypt registered [ 1532.324645] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1533.739722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1534.212355] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1534.403414] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1534.408302] LNet: Accept secure, port 988 [ 1536.064137] Key type lgssc registered [ 1537.228422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.778995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.353252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1546.411647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.612364] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1551.845220] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.972322] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.913850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.258236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1563.607512] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1571.687962] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1578.609261] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1585.306612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1585.628577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.570291] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1596.970738] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.957731] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 1603.044232] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1603.049617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.052485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1603.075750] Lustre: Skipped 1 previous similar message [ 1608.184770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1608.190346] Lustre: Skipped 1 previous similar message [ 1613.280220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1613.289642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.312200] Lustre: Skipped 1 previous similar message [ 1613.651920] Lustre: server umount lustre-OST0000 complete [ 1618.408975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.424955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.431429] Lustre: Skipped 1 previous similar message [ 1618.439611] Lustre: Skipped 2 previous similar messages [ 1623.523849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1623.534270] Lustre: Skipped 1 previous similar message [ 1623.705575] Lustre: server umount lustre-MDT0000 complete [ 1627.249053] LustreError: 53808:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189072 with bad export cookie 15253018078705522999 [ 1627.262923] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.285279] LustreError: 53808:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1627.730143] Lustre: server umount lustre-MDT0001 complete [ 1640.683349] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1643.487434] Key type lgssc unregistered [ 1643.780602] LNet: 57481:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.788371] LNet: Removed LNI 192.168.202.103@tcp [ 1644.532237] Key type .llcrypt unregistered [ 1644.534894] Key type ._llcrypt unregistered [ 1665.208546] alg: No test for adler32 (adler32-zlib) [ 1665.966196] Key type ._llcrypt registered [ 1665.968479] Key type .llcrypt registered [ 1666.076912] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1667.068164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.616701] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1667.804371] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1667.812370] LNet: Accept secure, port 988 [ 1669.480281] Key type lgssc registered [ 1670.406354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1677.121395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1694.752310] LustreError: 58916:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 1725.472994] LustreError: 58916:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 1755.233131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.513809] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1767.010301] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1773.275975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.201342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1778.324946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.448647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 1782.752728] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1783.786963] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1788.918363] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.749751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.092783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.673973] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1806.860928] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1813.161571] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.039479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.052778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1816.062109] Lustre: Skipped 1 previous similar message [ 1816.074230] Lustre: Skipped 1 previous similar message [ 1821.155187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.161311] Lustre: Skipped 1 previous similar message [ 1821.377789] Lustre: server umount lustre-OST0000 complete [ 1842.652862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.656069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.671048] Lustre: Skipped 1 previous similar message [ 1842.677242] Lustre: Skipped 1 previous similar message [ 1847.781707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.787455] Lustre: Skipped 1 previous similar message [ 1848.452116] Lustre: server umount lustre-MDT0000 complete [ 1852.417524] LustreError: 58923:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189297 with bad export cookie 4718520881651348422 [ 1852.433451] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.865064] Lustre: server umount lustre-MDT0001 complete [ 1869.309914] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 1871.477639] Key type lgssc unregistered [ 1871.673199] LNet: 62785:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.686304] LNet: Removed LNI 192.168.202.103@tcp [ 1872.413553] Key type .llcrypt unregistered [ 1872.416067] Key type ._llcrypt unregistered [ 1893.017053] alg: No test for adler32 (adler32-zlib) [ 1893.773848] Key type ._llcrypt registered [ 1893.775267] Key type .llcrypt registered [ 1893.879994] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 1894.918418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.624722] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 1895.846259] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 1895.852327] LNet: Accept secure, port 988 [ 1897.489252] Key type lgssc registered [ 1898.607448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.060234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.870866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.021555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.363405] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.797344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1920.485478] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.603567] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.570934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.046077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.181183] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1940.143651] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.261366] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.217906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.515579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.106833] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 1960.688943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1966.531323] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1983.457669] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.461427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.483096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.020546] Lustre: server umount lustre-OST0000 complete [ 1991.654438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.692089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.703948] Lustre: Skipped 1 previous similar message [ 1994.724370] Lustre: 67122:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776189434/real 1776189434] req@0000000066751654 x1862469514444800/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776189440 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1994.966080] Lustre: server umount lustre-MDT0000 complete [ 1996.778907] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.801852] LustreError: Skipped 1 previous similar message [ 1999.298563] LustreError: 65035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189444 with bad export cookie 3735526314333564781 [ 1999.302970] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.308406] LustreError: 65035:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1999.747683] Lustre: server umount lustre-MDT0001 complete [ 2012.145755] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2015.086372] Key type lgssc unregistered [ 2015.398606] LNet: 67926:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.428271] LNet: Removed LNI 192.168.202.103@tcp [ 2016.379813] Key type .llcrypt unregistered [ 2016.381762] Key type ._llcrypt unregistered [ 2058.960420] alg: No test for adler32 (adler32-zlib) [ 2059.722090] Key type ._llcrypt registered [ 2059.728638] Key type .llcrypt registered [ 2059.869309] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2061.516690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.210416] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2062.497816] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2062.507377] LNet: Accept secure, port 988 [ 2064.296241] Key type lgssc registered [ 2065.516237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2073.634358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.339619] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.426863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.742924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.443049] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2085.871471] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.978211] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2092.401115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.827688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.880593] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2107.586227] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2114.475060] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2122.429786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.723850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.811424] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 2127.826407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2136.362581] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2141.904896] Lustre: Failing over lustre-MDT0000 [ 2142.150172] Lustre: server umount lustre-MDT0000 complete [ 2142.798055] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.226041] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.236656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.287099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.307057] Lustre: Skipped 1 previous similar message [ 2151.395779] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.410076] LustreError: Skipped 8 previous similar messages [ 2152.802695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.979294] LustreError: 11-0: MGC192.168.202.103@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2152.988219] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.021061] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0xc7072e2c1508f030 to 0xc7072e2c1508f419 [ 2153.026306] LustreError: 69015:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 2153.040164] Lustre: MGC192.168.202.103@tcp: Connection restored to (at 0@lo) [ 2153.187412] Lustre: *** cfs_fail_loc=135, val=0*** [ 2153.301278] BUG: spinlock bad magic on CPU#2, llog_process_th/72860 [ 2153.303784] lock: 0xffff8d63c64984c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 2153.311295] CPU: 2 PID: 72860 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2153.322216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2153.327727] Call Trace: [ 2153.332136] ? dump_stack+0xbb/0x10e [ 2153.334230] ? spin_dump.cold.3+0x32/0x65 [ 2153.338428] ? do_raw_spin_lock+0xce/0x190 [ 2153.343014] ? _raw_spin_lock+0x21/0x30 [ 2153.346357] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 2153.348602] ? mdd_process_config+0x368/0x9c0 [mdd] [ 2153.350397] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 2153.353407] ? mdt_init0+0x650/0x1750 [mdt] [ 2153.355891] ? mdt_device_alloc+0x118/0x170 [mdt] [ 2153.359094] ? obd_setup+0x196/0x3e0 [obdclass] [ 2153.361415] ? class_setup+0x6ee/0x9e0 [obdclass] [ 2153.364220] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 2153.369412] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 2153.377595] ? llog_process_thread+0x138f/0x2270 [obdclass] [ 2153.382633] ? mgs_key_init+0xdd/0x110 [mgs] [ 2153.386568] ? keys_fill+0x10d/0x1b0 [obdclass] [ 2153.389600] ? lu_context_init+0xc8/0x240 [obdclass] [ 2153.394066] ? llog_process+0x40/0x40 [obdclass] [ 2153.397369] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 2153.401788] ? kthread+0x1d1/0x200 [ 2153.406256] ? set_kthread_struct+0x70/0x70 [ 2153.408459] ? ret_from_fork+0x1f/0x30 [ 2153.419549] LustreError: 72860:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 2153.427323] LustreError: 72860:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -2 [ 2153.435351] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2153.435351] [ 2153.451625] LustreError: 15c-8: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2153.464743] LustreError: 72848:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2153.472367] LustreError: 72848:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2153.483000] LustreError: 72848:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2153.522536] Lustre: server umount lustre-MDT0000 complete [ 2153.529138] LustreError: 72848:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2161.306341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.531907] LustreError: 11-0: MGC192.168.202.103@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2161.541812] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.564654] LustreError: 69695:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2161.569433] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0xc7072e2c1508f419 to 0xc7072e2c1508f4a5 [ 2161.589925] Lustre: MGC192.168.202.103@tcp: Connection restored to (at 0@lo) [ 2161.812201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.897485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2166.281760] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2167.279557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2167.304328] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2167.358772] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 2168.049348] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2178.028682] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.042640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.060513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.069792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.094192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.629565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.121207] Lustre: server umount lustre-OST0000 complete [ 2192.865740] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.869938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.876088] LustreError: Skipped 1 previous similar message [ 2192.881214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.881222] Lustre: Skipped 1 previous similar message [ 2192.917027] Lustre: Skipped 1 previous similar message [ 2195.289254] Lustre: server umount lustre-MDT0000 complete [ 2197.989273] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.008052] LustreError: Skipped 16 previous similar messages [ 2200.493926] LustreError: 72861:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189645 with bad export cookie 14341482305250981029 [ 2200.503173] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.507995] LustreError: 72861:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2201.047206] Lustre: server umount lustre-MDT0001 complete [ 2214.802732] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2217.908504] Key type lgssc unregistered [ 2218.158061] LNet: 74761:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.164372] LNet: Removed LNI 192.168.202.103@tcp [ 2218.987270] Key type .llcrypt unregistered [ 2218.995023] Key type ._llcrypt unregistered [ 2245.435125] alg: No test for adler32 (adler32-zlib) [ 2246.194982] Key type ._llcrypt registered [ 2246.196499] Key type .llcrypt registered [ 2246.318622] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2247.421420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2248.427432] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2248.704434] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2248.718314] LNet: Accept secure, port 988 [ 2250.448196] Key type lgssc registered [ 2251.726840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.212950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.779287] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.874582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.107796] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.257319] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2272.239873] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.319108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.350911] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.739198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.063680] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2292.023965] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2299.554964] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2305.609497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2305.893486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.002434] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2312.071878] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 2319.231853] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2324.907487] Lustre: Failing over lustre-MDT0000 [ 2325.121844] Lustre: server umount lustre-MDT0000 complete [ 2326.080885] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.531129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.545116] Lustre: Skipped 2 previous similar messages [ 2334.698601] Lustre: 75556:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776189772/real 1776189772] req@0000000062c0331c x1862469884588992/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776189779 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2334.747614] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.785868] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.818046] LustreError: Skipped 10 previous similar messages [ 2336.953662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.844596] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0xd57b3a85d11c4dbe to 0xd57b3a85d11c51ae [ 2340.854226] Lustre: MGC192.168.202.103@tcp: Connection restored to (at 0@lo) [ 2340.958953] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2341.030283] LustreError: 79402:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 2341.035704] LustreError: 79402:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -9 [ 2341.040583] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2341.040583] [ 2341.044694] LustreError: 15c-8: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2341.051900] LustreError: 79389:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2341.057388] LustreError: 79389:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 2341.064751] LustreError: 79389:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2341.087614] Lustre: server umount lustre-MDT0000 complete [ 2341.089824] LustreError: 79389:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2350.715459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.086221] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.129263] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.141916] LustreError: Skipped 15 previous similar messages [ 2351.204794] LustreError: 76236:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2351.221474] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0xd57b3a85d11c51ae to 0xd57b3a85d11c5248 [ 2351.263578] Lustre: MGC192.168.202.103@tcp: Connection restored to (at 0@lo) [ 2351.558932] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.3@tcp (not set up) [ 2351.677917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.737988] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2356.195520] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2356.726147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2356.726827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2356.800796] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2356.915543] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 2366.946655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.955962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.968350] Lustre: Skipped 1 previous similar message [ 2366.994580] Lustre: Skipped 1 previous similar message [ 2372.065328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.071788] Lustre: Skipped 1 previous similar message [ 2372.253484] Lustre: server umount lustre-OST0000 complete [ 2377.186404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.187053] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2377.216569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.242600] Lustre: Skipped 1 previous similar message [ 2382.305714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.313224] Lustre: Skipped 1 previous similar message [ 2383.170307] Lustre: server umount lustre-MDT0000 complete [ 2387.425522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.425522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.667254] LustreError: 76225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189832 with bad export cookie 15382953298627416648 [ 2387.670666] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.681569] LustreError: 76225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2388.043185] Lustre: server umount lustre-MDT0001 complete [ 2405.476218] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2408.932891] Key type lgssc unregistered [ 2409.395592] LNet: 81290:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.416147] LNet: Removed LNI 192.168.202.103@tcp [ 2410.374349] Key type .llcrypt unregistered [ 2410.378329] Key type ._llcrypt unregistered [ 2436.134271] alg: No test for adler32 (adler32-zlib) [ 2436.895910] Key type ._llcrypt registered [ 2436.898104] Key type .llcrypt registered [ 2437.051486] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2438.432453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.866877] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2439.105987] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2439.125113] LNet: Accept secure, port 988 [ 2440.800180] Key type lgssc registered [ 2442.085333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2450.277658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.912270] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.996655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.058063] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.601317] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2462.183467] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2467.305829] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.055619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.713282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.312943] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2483.625425] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.644599] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.861077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.173140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.264130] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 2503.044867] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2511.375170] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.106483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.133862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.482075] Lustre: server umount lustre-OST0000 complete [ 2533.348257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.354087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.362149] Lustre: Skipped 2 previous similar messages [ 2533.377879] Lustre: Skipped 2 previous similar messages [ 2538.037107] Lustre: server umount lustre-MDT0000 complete [ 2538.475489] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.484708] LustreError: Skipped 1 previous similar message [ 2542.049248] LustreError: 82734:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189987 with bad export cookie 3840210718947563346 [ 2542.053634] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.064878] LustreError: 82734:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2542.434549] Lustre: server umount lustre-MDT0001 complete [ 2556.735683] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2561.057600] Key type lgssc unregistered [ 2561.302496] LNet: 86413:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.322850] LNet: Removed LNI 192.168.202.103@tcp [ 2562.226595] Key type .llcrypt unregistered [ 2562.241297] Key type ._llcrypt unregistered [ 2585.476528] alg: No test for adler32 (adler32-zlib) [ 2586.229807] Key type ._llcrypt registered [ 2586.231903] Key type .llcrypt registered [ 2586.306979] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2587.462470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2588.043500] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2588.242804] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2588.252159] LNet: Accept secure, port 988 [ 2589.928137] Key type lgssc registered [ 2591.081797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2599.537237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.221186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.296269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.149628] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2606.563227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.681443] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.625243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.792747] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.050814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.985164] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2631.048448] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2638.556798] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2646.656278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.290650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.375893] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 2652.655577] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2661.425541] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.075067] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2671.087223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.112331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.202881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.213390] Lustre: Skipped 1 previous similar message [ 2676.431713] Lustre: server umount lustre-OST0000 complete [ 2681.314700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.317664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.331637] Lustre: Skipped 2 previous similar messages [ 2681.336110] Lustre: Skipped 1 previous similar message [ 2686.438986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.451443] Lustre: Skipped 1 previous similar message [ 2686.853105] Lustre: server umount lustre-MDT0000 complete [ 2690.548767] LustreError: 87860:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190135 with bad export cookie 13365881853517248368 [ 2690.558856] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.565020] LustreError: 87860:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2690.899586] Lustre: server umount lustre-MDT0001 complete [ 2704.701982] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2707.707468] Key type lgssc unregistered [ 2708.148525] LNet: 91541:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.162749] LNet: Removed LNI 192.168.202.103@tcp [ 2709.257789] Key type .llcrypt unregistered [ 2709.263776] Key type ._llcrypt unregistered [ 2735.324905] alg: No test for adler32 (adler32-zlib) [ 2736.083206] Key type ._llcrypt registered [ 2736.086942] Key type .llcrypt registered [ 2736.277201] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2737.800916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2738.593513] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2738.897530] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2738.902813] LNet: Accept secure, port 988 [ 2740.640137] Key type lgssc registered [ 2742.048064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2752.618838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.117369] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.186743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.653144] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.066646] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2764.776058] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.884698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.986484] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.130887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.651400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2779.968739] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2789.164771] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.442941] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.328379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2804.801936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.858493] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 2810.401190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 2820.573404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2840.549066] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.559216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.576111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2840.576111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.875415] Lustre: server umount lustre-OST0000 complete [ 2850.788357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.798691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.809510] Lustre: Skipped 2 previous similar messages [ 2850.826317] Lustre: Skipped 1 previous similar message [ 2852.913794] Lustre: server umount lustre-MDT0000 complete [ 2855.908964] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.924831] LustreError: Skipped 1 previous similar message [ 2857.652680] LustreError: 92984:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190302 with bad export cookie 11798899236420797325 [ 2857.656860] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.663254] LustreError: 92984:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2858.029757] Lustre: server umount lustre-MDT0001 complete [ 2871.991427] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 2874.925088] Key type lgssc unregistered [ 2875.252991] LNet: 96686:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.260749] LNet: Removed LNI 192.168.202.103@tcp [ 2876.084641] Key type .llcrypt unregistered [ 2876.087923] Key type ._llcrypt unregistered [ 2899.845328] alg: No test for adler32 (adler32-zlib) [ 2900.613996] Key type ._llcrypt registered [ 2900.617995] Key type .llcrypt registered [ 2900.786756] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 2902.209570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.059414] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 2903.304366] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 2903.310951] LNet: Accept secure, port 988 [ 2904.992184] Key type lgssc registered [ 2906.533462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2917.279483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.008468] LustreError: 98124:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 2966.752214] LustreError: 98124:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 2996.512328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.188761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3012.556376] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3018.615355] Lustre: server umount lustre-OST0000 complete [ 3028.090927] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:14:31 (1776190471) [ 3035.838077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.543952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.608076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.434251] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3041.765481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.885181] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.005284] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.495821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.226053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3059.320607] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3069.813049] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3077.311948] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3085.008376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.318747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.361490] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 3090.778827] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3100.113334] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3118.051683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.068897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.076863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.081823] Lustre: Skipped 1 previous similar message [ 3122.717682] Lustre: server umount lustre-OST0000 complete [ 3128.291472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.305148] Lustre: Skipped 1 previous similar message [ 3128.309767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.410166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.429902] Lustre: Skipped 2 previous similar messages [ 3133.619846] Lustre: server umount lustre-MDT0000 complete [ 3134.433792] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.463140] LustreError: Skipped 1 previous similar message [ 3138.242922] LustreError: 99707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190583 with bad export cookie 15744961731073416704 [ 3138.251058] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3138.254150] LustreError: 99707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3138.619781] Lustre: server umount lustre-MDT0001 complete [ 3151.747724] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3154.232328] Key type lgssc unregistered [ 3154.562821] LNet: 103511:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3154.573798] LNet: Removed LNI 192.168.202.103@tcp [ 3155.507953] Key type .llcrypt unregistered [ 3155.510494] Key type ._llcrypt unregistered [ 3179.202699] alg: No test for adler32 (adler32-zlib) [ 3179.962944] Key type ._llcrypt registered [ 3179.966410] Key type .llcrypt registered [ 3180.099841] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3181.621792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3182.498480] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 3182.784549] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3182.795705] LNet: Accept secure, port 988 [ 3184.592146] Key type lgssc registered [ 3185.851409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3196.371906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.074481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.162990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.561113] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.662386] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3208.678381] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.777783] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.309973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.666281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.362655] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3235.939286] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3246.929951] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3257.104080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.828692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.882258] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3273.160757] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 3274.164766] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3286.498636] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3286.505401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.511792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.554500] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3288.586801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.587862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.644099] Lustre: Skipped 1 previous similar message [ 3290.366906] Lustre: server umount lustre-OST0000 complete [ 3297.764866] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3297.769161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.791895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.408219] Lustre: server umount lustre-MDT0000 complete [ 3303.910469] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.941310] LustreError: Skipped 1 previous similar message [ 3307.620056] LustreError: 104962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190752 with bad export cookie 10766647873404300377 [ 3307.634289] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3307.640624] LustreError: 104962:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3308.133497] Lustre: server umount lustre-MDT0001 complete [ 3323.144121] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 3327.643878] Key type lgssc unregistered [ 3328.070895] LNet: 108645:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3328.087290] LNet: Removed LNI 192.168.202.103@tcp [ 3329.361613] Key type .llcrypt unregistered [ 3329.367300] Key type ._llcrypt unregistered [ 3349.333838] alg: No test for adler32 (adler32-zlib) [ 3350.095034] Key type ._llcrypt registered [ 3350.101046] Key type .llcrypt registered [ 3350.199371] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3351.609733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.644934] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 3353.009474] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 3353.015641] LNet: Accept secure, port 988 [ 3354.779989] Key type lgssc registered [ 3356.409986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.573892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.264864] LustreError: 109971:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 3415.011767] LustreError: 109971:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 3445.792381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.251268] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3460.992366] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3471.119306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.937419] LustreError: 15c-8: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3491.961768] LustreError: 110961:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3491.986957] LustreError: 110961:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 3492.009573] LustreError: 110961:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 3492.012853] LustreError: 109979:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190937 with bad export cookie 6331933110095486208 [ 3492.031227] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.317122] Lustre: server umount lustre-MDT0000 complete [ 3492.320296] LustreError: 110961:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3519.247449] Lustre: server umount lustre-OST0000 complete [ 3533.739351] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3547.533971] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3559.587837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.231297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3577.233557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3585.152135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.268865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.364995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.561848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3593.596706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3593.654237] Lustre: lustre-MDT0000: new disk, initializing [ 3593.737892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.763862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3599.229736] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3616.088425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3616.252073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.435558] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3616.508813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3616.521840] Lustre: Skipped 1 previous similar message [ 3616.729801] Lustre: lustre-MDT0001: new disk, initializing [ 3616.982821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3617.068936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3617.086606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3623.909670] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3637.312123] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.222995] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3654.082301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3654.186836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.194489] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3654.739970] Lustre: lustre-OST0000: new disk, initializing [ 3654.749193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3654.896554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3660.931860] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3661.440508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3661.451660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3673.168775] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3680.722686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3680.964313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3687.386675] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3687.911520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3692.528072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.530560] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3692.533290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3692.565211] Lustre: Skipped 1 previous similar message [ 3692.617244] LustreError: Skipped 1 previous similar message [ 3696.927635] Lustre: server umount lustre-OST0000 complete [ 3702.754489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.756407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.764944] Lustre: Skipped 1 previous similar message [ 3702.775361] Lustre: Skipped 2 previous similar messages [ 3707.235285] Lustre: server umount lustre-MDT0000 complete [ 3707.874761] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.907782] LustreError: Skipped 1 previous similar message [ 3711.730895] LustreError: 114704:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191157 with bad export cookie 6331933110095486299 [ 3711.732022] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.744191] LustreError: 114704:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3712.051979] Lustre: server umount lustre-MDT0001 complete [ 3722.094534] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:26:06 (1776191166) [ 3725.707540] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3743.654594] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 3755.741949] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 3769.372957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3778.179042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3785.062420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3792.543557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3800.780069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3800.863975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.117868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3801.163539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3801.258607] Lustre: lustre-MDT0000: new disk, initializing [ 3801.370584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.388899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3807.215483] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3819.918906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3820.042272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.167284] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3820.260399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3820.267566] Lustre: Skipped 1 previous similar message [ 3820.401593] Lustre: lustre-MDT0001: new disk, initializing [ 3820.535785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.579320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3820.590715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3826.675910] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3837.938923] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.974376] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3851.787597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.847246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.851917] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3852.016388] Lustre: lustre-OST0000: new disk, initializing [ 3852.020034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3852.086368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.324953] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3857.512850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3857.533712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3868.868627] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3873.915863] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3874.171198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3879.283969] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3879.570332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3883.491920] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3883.493921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.497555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3883.506582] Lustre: Skipped 1 previous similar message [ 3883.518062] Lustre: Skipped 1 previous similar message [ 3887.882753] Lustre: server umount lustre-OST0000 complete [ 3892.199735] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3892.211544] LustreError: Skipped 1 previous similar message [ 3892.216629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.232714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.849675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.881359] LustreError: Skipped 1 previous similar message [ 3903.087664] LustreError: 121845:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191348 with bad export cookie 6331933110095487146 [ 3903.092373] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3903.103785] LustreError: 121845:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3913.518737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.043268] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.100967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.333770] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.062996] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3924.461737] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.328772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.579864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3929.588339] Lustre: Skipped 2 previous similar messages [ 3934.722991] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3943.683598] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.244486] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3957.282531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3962.994468] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 3973.269565] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3980.903940] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3989.485805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.497956] Lustre: Skipped 2 previous similar messages [ 3989.510727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3989.518732] Lustre: Skipped 1 previous similar message [ 3992.566516] Lustre: server umount lustre-OST0000 complete [ 3992.573863] Lustre: Skipped 2 previous similar messages [ 3999.715206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.839618] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.863396] LustreError: Skipped 1 previous similar message [ 4006.981638] LustreError: 128301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191452 with bad export cookie 6331933110095488007 [ 4006.982872] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.991284] LustreError: 128301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4020.818564] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 4023.649425] Key type lgssc unregistered [ 4023.989672] LNet: 129105:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4024.004228] LNet: Removed LNI 192.168.202.103@tcp [ 4024.866216] Key type .llcrypt unregistered [ 4024.871795] Key type ._llcrypt unregistered [ 4048.501614] alg: No test for adler32 (adler32-zlib) [ 4049.258528] Key type ._llcrypt registered [ 4049.260878] Key type .llcrypt registered [ 4049.383686] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 4063.834521] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 4064.710865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4065.613051] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 4065.876567] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 4065.886982] LNet: Accept secure, port 988 [ 4067.578146] Key type lgssc registered [ 4068.622785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4078.673161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4086.894638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.536115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.322720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.636875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.742649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.170051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4114.211985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4114.379970] Lustre: lustre-MDT0000: new disk, initializing [ 4114.489289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.515973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.205326] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4132.964138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4133.054513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.259214] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4133.324106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4133.336170] Lustre: Skipped 1 previous similar message [ 4133.437551] Lustre: lustre-MDT0001: new disk, initializing [ 4133.637906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4133.708818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4133.729912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4140.369731] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4152.879331] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4159.156885] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4167.480125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.587424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.597517] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4167.969170] Lustre: lustre-OST0000: new disk, initializing [ 4167.973824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4168.095255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.530911] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4178.024797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.034708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4184.630944] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4191.020337] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4191.378398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4199.434301] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4199.759205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4204.003116] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4204.007444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.008986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.009465] LustreError: Skipped 1 previous similar message [ 4204.024163] Lustre: Skipped 1 previous similar message [ 4208.454934] Lustre: server umount lustre-OST0000 complete [ 4214.243249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.245550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.269116] Lustre: Skipped 1 previous similar message [ 4214.280156] Lustre: Skipped 1 previous similar message [ 4219.363706] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.388817] LustreError: Skipped 1 previous similar message [ 4219.457834] Lustre: server umount lustre-MDT0000 complete [ 4220.395603] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.426046] LustreError: Skipped 1 previous similar message [ 4223.972964] LustreError: 132393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191669 with bad export cookie 8742015829526473513 [ 4223.976981] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.988330] LustreError: 132393:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4224.233492] Lustre: server umount lustre-MDT0001 complete [ 4234.319836] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:34:37 (1776191677) [ 4241.196662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.558071] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.621370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.052680] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4247.014863] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.145712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.131760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.646168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4260.386653] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4268.672730] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4275.829789] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4281.313264] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4281.320130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.335907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.341535] Lustre: Skipped 1 previous similar message [ 4283.362686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.366448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.637689] Lustre: server umount lustre-MDT0000 complete [ 4288.488604] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.498475] LustreError: Skipped 1 previous similar message [ 4288.977242] LustreError: 136936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191734 with bad export cookie 8742015829526474360 [ 4288.985328] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.361762] Lustre: server umount lustre-MDT0001 complete [ 4299.707261] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:35:43 (1776191743) [ 4307.735599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.408630] LustreError: 139478:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 4336.736432] LustreError: 139478:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4365.537585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.527521] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4377.947879] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.021977] Lustre: server umount lustre-OST0000 complete [ 4388.041962] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:37:12 (1776191832) [ 4395.231319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.838926] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.979968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.037239] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4409.900348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.324346] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4424.165890] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4431.938185] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4439.015400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4439.324593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.341649] Lustre: Skipped 1 previous similar message [ 4445.693209] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4455.760289] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4466.659389] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4466.670245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.685660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4466.691679] Lustre: Skipped 1 previous similar message [ 4472.019427] Lustre: server umount lustre-MDT0000 complete [ 4472.297730] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.324816] LustreError: Skipped 4 previous similar messages [ 4476.860911] LustreError: 140866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191922 with bad export cookie 8742015829526475032 [ 4476.872903] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.882775] LustreError: 140866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4488.162685] Lustre: 143940:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776191927/real 1776191927] req@0000000081eea3ea x1862471789886912/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776191933 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4498.106357] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:39:02 (1776191942) [ 4505.690734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.349498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.812553] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4521.541366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.669580] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4537.605691] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4544.212256] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4551.647194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.305422] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 4557.614274] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4565.369816] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4570.656892] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4570.994258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4576.271651] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4576.581154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4582.913405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.917566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.940249] Lustre: Skipped 3 previous similar messages [ 4582.942738] Lustre: Skipped 3 previous similar messages [ 4582.962706] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.126835] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4593.627779] Lustre: server umount lustre-OST0000 complete [ 4593.642880] Lustre: Skipped 2 previous similar messages [ 4603.363759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4603.369635] Lustre: Skipped 8 previous similar messages [ 4608.481962] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.503456] LustreError: Skipped 4 previous similar messages [ 4608.680811] LustreError: 144601:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192053 with bad export cookie 8742015829526476103 [ 4608.683140] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4608.687331] LustreError: 144601:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4618.020363] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:41:02 (1776192062) [ 4624.984965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.592807] LustreError: 148950:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 4653.856773] LustreError: 148950:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4683.680431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.684419] Lustre: Skipped 2 previous similar messages [ 4688.408208] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4696.160576] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4702.768702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.840117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 4730.886929] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4740.638702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.941174] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4755.193764] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4760.865994] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4766.775257] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4767.037829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4773.314864] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4773.575880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4776.931916] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.943875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.952868] Lustre: Skipped 2 previous similar messages [ 4776.963598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4790.241146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4790.458499] Lustre: server umount lustre-MDT0000 complete [ 4790.472309] Lustre: Skipped 2 previous similar messages [ 4793.320737] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.358766] LustreError: Skipped 5 previous similar messages [ 4797.223166] LustreError: 148958:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192242 with bad export cookie 8742015829526477132 [ 4797.229711] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4809.184218] Lustre: 152671:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192248/real 1776192248] req@00000000b3519b5e x1862471789921216/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192254 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4822.678961] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:44:26 (1776192266) [ 4830.640935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4847.201266] LustreError: 153316:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 4858.466262] LustreError: 153316:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 4892.791708] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4902.557710] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4910.356897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.651915] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 4937.664407] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4949.066652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.669991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4949.676594] Lustre: Skipped 4 previous similar messages [ 4954.869456] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4963.077978] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4969.056395] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4975.759586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.849893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4975.860808] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4976.155369] Lustre: lustre-OST0001: new disk, initializing [ 4976.162347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4981.159769] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 4982.379352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4982.388223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4991.347946] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4996.222453] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4996.473967] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5001.071288] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5001.304520] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5005.810274] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5005.817390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.832147] Lustre: Skipped 2 previous similar messages [ 5005.839554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5005.844500] Lustre: Skipped 9 previous similar messages [ 5009.476590] Lustre: server umount lustre-OST0000 complete [ 5009.482389] Lustre: Skipped 2 previous similar messages [ 5027.298469] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5031.393378] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5031.401025] LustreError: Skipped 2 previous similar messages [ 5041.193963] LustreError: 153323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192486 with bad export cookie 8742015829526478077 [ 5041.203415] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.213959] LustreError: 153323:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5048.284812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.725197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5057.352894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.802669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5067.736771] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:48:31 (1776192511) [ 5069.440333] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5071.226553] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5073.018941] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:48:37 (1776192517) [ 5080.115088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.672766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5080.719690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5085.512547] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5094.297920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.367202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5094.414222] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5099.222879] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5110.820331] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5120.235939] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5128.034994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.167666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5131.785882] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5134.144238] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 5139.550422] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5145.519043] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5145.705567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5150.316257] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5150.536549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5154.789265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.798357] Lustre: Skipped 5 previous similar messages [ 5154.800323] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5154.800789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5154.809052] Lustre: Skipped 15 previous similar messages [ 5181.943638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.004670] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5187.383070] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.414742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.429914] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 5187.433854] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 5187.465500] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 5197.874128] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5205.085635] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5205.362962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5208.035275] Lustre: lustre-MDT0000: haven't heard from client db28d9e8-516c-4cc5-a233-951e6ed6bd0e (at 192.168.202.3@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp 000000009570325d, cur 1776192653 expire 1776192623 last 1776192606 [ 5211.578664] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5211.859234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5218.398430] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8fcc0312e000.ost_server_uuid 40 [ 5220.354804] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8fcc0312e000.ost_server_uuid in FULL state after 0 sec [ 5238.241483] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5243.875344] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.899824] LustreError: Skipped 13 previous similar messages [ 5249.101818] LustreError: 159892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192694 with bad export cookie 8742015829526479232 [ 5249.110159] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.142868] LustreError: 159892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5261.673561] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5263.861225] Key type lgssc unregistered [ 5264.119171] LNet: 166301:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5264.123807] LNet: Removed LNI 192.168.202.103@tcp [ 5264.815580] Key type .llcrypt unregistered [ 5264.818185] Key type ._llcrypt unregistered [ 5292.230128] alg: No test for adler32 (adler32-zlib) [ 5293.011700] Key type ._llcrypt registered [ 5293.014181] Key type .llcrypt registered [ 5293.274970] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5294.838857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5296.060120] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 5296.395404] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5296.412873] LNet: Accept secure, port 988 [ 5298.344201] Key type lgssc registered [ 5300.223468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5309.989714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.359111] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.456688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.598342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5317.278190] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5321.703397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.686148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.812234] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5326.080665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5330.081335] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5338.753102] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5348.683573] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5357.976679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.511290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.610405] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 5365.593746] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5376.538306] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5384.084518] Lustre: Failing over lustre-MDT0000 [ 5384.428833] Lustre: server umount lustre-MDT0000 complete [ 5386.214934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.223034] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.231509] Lustre: Skipped 2 previous similar messages [ 5386.257182] LustreError: Skipped 2 previous similar messages [ 5393.440890] Lustre: 167097:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192831/real 1776192831] req@00000000ae7974f5 x1862473079601216/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776192838 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5393.459168] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5398.498801] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.507512] LustreError: Skipped 8 previous similar messages [ 5405.854249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.798026] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0xec3d58a68611946a to 0xec3d58a68611984c [ 5410.824930] Lustre: MGC192.168.202.103@tcp: Connection restored to 192.168.202.103@tcp (at 0@lo) [ 5411.202426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.273361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5416.418377] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5416.452917] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5417.800831] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5421.142303] Lustre: lustre-MDT0000: Denying connection for new client 4dead3d2-723c-4f85-9420-e6e539341b63 (at 192.168.202.3@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5431.797110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.800145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.812771] Lustre: Skipped 1 previous similar message [ 5431.833042] Lustre: Skipped 1 previous similar message [ 5436.099772] Lustre: server umount lustre-OST0000 complete [ 5439.873688] LustreError: 171642:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5439.879557] Lustre: 170939:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5439.883962] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5439.900771] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5439.916825] LustreError: 170939:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002be946b4 x1862473079612288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [ 5439.925492] LustreError: 170939:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5439.930844] LustreError: 170939:0:(fid_request.c:335:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5439.939898] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5439.983719] Lustre: 170939:0:(mdt_handler.c:7538:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 5440.145566] Lustre: server umount lustre-MDT0000 complete [ 5442.017730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.030648] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.053743] LustreError: Skipped 12 previous similar messages [ 5445.186617] LustreError: 167094:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5445.190168] LustreError: 168572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192890 with bad export cookie 17022859638767458380 [ 5445.195637] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5445.227736] LustreError: 168572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5445.578972] Lustre: server umount lustre-MDT0001 complete [ 5458.939234] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5462.415069] Key type lgssc unregistered [ 5462.613700] LNet: 172447:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5462.619570] LNet: Removed LNI 192.168.202.103@tcp [ 5463.379026] Key type .llcrypt unregistered [ 5463.385622] Key type ._llcrypt unregistered [ 5488.299596] alg: No test for adler32 (adler32-zlib) [ 5489.057243] Key type ._llcrypt registered [ 5489.062190] Key type .llcrypt registered [ 5489.190490] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5490.229348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5490.906745] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 5491.150856] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5491.154941] LNet: Accept secure, port 988 [ 5492.944308] Key type lgssc registered [ 5494.273414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5502.630766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.151188] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5504.246586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.300605] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5509.610480] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5514.736869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.831422] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.738732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.055712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.919759] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5534.918234] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5541.675770] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5549.974984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.209960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.373122] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 5555.990866] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5565.192072] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5572.070629] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5572.079977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.103696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5575.144268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.154423] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.162905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.293850] Lustre: server umount lustre-OST0000 complete [ 5585.381216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.395544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.416465] Lustre: Skipped 1 previous similar message [ 5585.435129] Lustre: Skipped 2 previous similar messages [ 5589.006536] Lustre: server umount lustre-MDT0000 complete [ 5590.497298] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.531136] LustreError: Skipped 1 previous similar message [ 5593.509749] LustreError: 176764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193038 with bad export cookie 14984792209932875357 [ 5593.511238] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5593.527887] LustreError: 176764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5593.960735] Lustre: server umount lustre-MDT0001 complete [ 5606.700322] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5610.204826] Key type lgssc unregistered [ 5610.542573] LNet: 177568:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5610.559170] LNet: Removed LNI 192.168.202.103@tcp [ 5611.573993] Key type .llcrypt unregistered [ 5611.580225] Key type ._llcrypt unregistered [ 5645.409787] alg: No test for adler32 (adler32-zlib) [ 5646.161756] Key type ._llcrypt registered [ 5646.163983] Key type .llcrypt registered [ 5646.279344] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5647.251186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5647.990298] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 5648.217564] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5648.221777] LNet: Accept secure, port 988 [ 5649.912147] Key type lgssc registered [ 5650.930744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5658.718413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.113082] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.149680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.370886] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5665.254538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.372462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.126741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.496446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.956858] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5688.173492] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5695.161472] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5703.242221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5703.743917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.307624] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5709.872621] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 5718.617404] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5730.273592] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.289375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.302083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.347471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.355794] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5733.360780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.023919] Lustre: server umount lustre-OST0000 complete [ 5743.585842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.587027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.605823] Lustre: Skipped 1 previous similar message [ 5743.611035] Lustre: Skipped 2 previous similar messages [ 5748.706675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.721790] Lustre: Skipped 1 previous similar message [ 5754.850489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.855042] Lustre: Skipped 3 previous similar messages [ 5755.360217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5755.460142] Lustre: server umount lustre-MDT0000 complete [ 5759.982774] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.009850] LustreError: Skipped 1 previous similar message [ 5760.070384] LustreError: 179303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193205 with bad export cookie 9114006742007302879 [ 5760.079582] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.086530] LustreError: 179303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5760.558410] Lustre: server umount lustre-MDT0001 complete [ 5773.599376] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5777.195757] Key type lgssc unregistered [ 5777.603853] LNet: 182984:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5777.614964] LNet: Removed LNI 192.168.202.103@tcp [ 5778.485640] Key type .llcrypt unregistered [ 5778.488801] Key type ._llcrypt unregistered [ 5801.156963] alg: No test for adler32 (adler32-zlib) [ 5801.922315] Key type ._llcrypt registered [ 5801.933562] Key type .llcrypt registered [ 5802.155794] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5803.832271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5804.342625] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 5804.711882] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5804.716827] LNet: Accept secure, port 988 [ 5806.552099] Key type lgssc registered [ 5807.479946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5822.296594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.793257] Lustre: *** cfs_fail_loc=135, val=0*** [ 5823.888757] LustreError: 184570:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 5823.893360] LustreError: 184570:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.103@tcp: cfg command failed: rc = -2 [ 5823.906196] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5823.906196] [ 5823.915702] LustreError: 15c-8: MGC192.168.202.103@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5823.938206] LustreError: 184531:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5823.946637] LustreError: 184531:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 5823.960094] LustreError: 184531:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 5830.114796] Lustre: 184531:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193269/real 1776193269] req@0000000022962f9f x1862473613312832/t0(0) o251->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193275 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5830.432101] Lustre: server umount lustre-MDT0000 complete [ 5830.435171] LustreError: 184531:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5839.160530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.770708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.869528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.965379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.653528] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5850.103188] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.217815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.620241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.004544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5860.130132] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 5868.260829] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5876.459832] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5881.825428] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5881.844407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.849877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5881.861510] Lustre: Skipped 1 previous similar message [ 5885.348165] Lustre: server umount lustre-MDT0000 complete [ 5885.987484] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.017722] LustreError: Skipped 1 previous similar message [ 5890.091140] LustreError: 186622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193335 with bad export cookie 10640995239460698535 [ 5890.101621] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5890.106691] LustreError: 186622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5890.386732] Lustre: server umount lustre-MDT0001 complete [ 5904.000647] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5907.295348] Key type lgssc unregistered [ 5907.611171] LNet: 187426:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5907.625945] LNet: Removed LNI 192.168.202.103@tcp [ 5908.768755] Key type .llcrypt unregistered [ 5908.771184] Key type ._llcrypt unregistered [ 5938.902305] alg: No test for adler32 (adler32-zlib) [ 5939.664166] Key type ._llcrypt registered [ 5939.666100] Key type .llcrypt registered [ 5939.816956] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 5940.120531] Key type .llcrypt unregistered [ 5940.123101] Key type ._llcrypt unregistered [ 5958.241670] alg: No test for adler32 (adler32-zlib) [ 5959.001959] Key type ._llcrypt registered [ 5959.012874] Key type .llcrypt registered [ 5959.122967] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 5960.709981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5961.434433] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 5961.728182] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 5961.732637] LNet: Accept secure, port 988 [ 5963.449135] Key type lgssc registered [ 5964.997757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5974.176777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5991.968386] LustreError: 189701:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 1: rc = -5 [ 6022.688601] LustreError: 189701:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.103@tcp: failed processing log, type 4: rc = -110 [ 6052.512541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.443989] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6066.462548] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6075.698404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.994777] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.212755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.337531] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 6105.590414] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.626361] LustreError: Skipped 1 previous similar message [ 6106.678618] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6110.693658] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.811516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.833569] LustreError: Skipped 1 previous similar message [ 6116.592496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.920636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6121.361494] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6129.415062] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6136.932645] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6140.575924] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6156.779564] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6156.795996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.816918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.802240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6157.810111] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.811363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.824537] Lustre: Skipped 1 previous similar message [ 6162.918066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.920602] Lustre: Skipped 1 previous similar message [ 6168.037350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.049858] Lustre: Skipped 1 previous similar message [ 6170.080143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6170.179303] Lustre: server umount lustre-OST0000 complete [ 6178.278623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.279527] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6178.289847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.303429] Lustre: Skipped 1 previous similar message [ 6181.105063] Lustre: server umount lustre-MDT0000 complete [ 6183.395306] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.411863] LustreError: Skipped 1 previous similar message [ 6185.709799] LustreError: 189708:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193631 with bad export cookie 16827000640006404216 [ 6185.712927] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.730692] LustreError: 189708:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6186.064611] Lustre: server umount lustre-MDT0001 complete [ 6198.255057] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6201.556904] Key type lgssc unregistered [ 6201.839613] LNet: 193850:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.851190] LNet: Removed LNI 192.168.202.103@tcp [ 6203.028258] Key type .llcrypt unregistered [ 6203.031071] Key type ._llcrypt unregistered [ 6229.586928] alg: No test for adler32 (adler32-zlib) [ 6230.342255] Key type ._llcrypt registered [ 6230.345913] Key type .llcrypt registered [ 6230.477381] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6231.919358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6233.277140] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 6233.552971] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6233.559590] LNet: Accept secure, port 988 [ 6235.336136] Key type lgssc registered [ 6236.804251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6244.961844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.499254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.614905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.001273] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.745804] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6257.124062] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.246347] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.346989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.738690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6268.124374] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6278.858436] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6285.853791] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6293.989692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.416440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.526334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 6300.300288] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6310.335503] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6315.577087] Lustre: Failing over lustre-MDT0000 [ 6315.818651] Lustre: server umount lustre-MDT0000 complete [ 6317.109414] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.052984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.071944] Lustre: Skipped 2 previous similar messages [ 6325.155363] Lustre: 194645:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193763/real 1776193763] req@00000000d07f0992 x1862474062116096/t0(0) o400->MGC192.168.202.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193770 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6325.189505] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6325.215690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.235731] LustreError: Skipped 10 previous similar messages [ 6337.904232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.602599] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.614678] LustreError: Skipped 15 previous similar messages [ 6342.635245] Lustre: Evicted from MGS (at 192.168.202.103@tcp) after server handle changed from 0x1b26f2857279608d to 0x1b26f2857279646f [ 6342.659393] Lustre: MGC192.168.202.103@tcp: Connection restored to 192.168.202.103@tcp (at 0@lo) [ 6342.977929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.039772] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6345.739484] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6348.271457] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 6348.293071] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6348.339600] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 6348.492585] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6353.600771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 6357.657222] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 6367.235333] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6367.243295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.268653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6368.565705] Lustre: server umount lustre-OST0000 complete [ 6373.862955] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6373.867886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6373.870559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6373.908851] Lustre: Skipped 1 previous similar message [ 6375.904445] Lustre: 194646:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193814/real 1776193814] req@00000000c8bf810a x1862474062132224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776193821 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 6375.944522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.958619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6375.970193] Lustre: Skipped 2 previous similar messages [ 6379.478278] Lustre: server umount lustre-MDT0000 complete [ 6381.024462] Lustre: 194646:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193819/real 1776193819] req@000000001d4d62dc x1862474062132864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776193826 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6381.027074] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.048462] Lustre: 194646:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6381.084221] LustreError: Skipped 4 previous similar messages [ 6384.020699] LustreError: 195313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193829 with bad export cookie 1956517743104189551 [ 6384.036344] LustreError: 195313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6384.038257] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.503210] Lustre: server umount lustre-MDT0001 complete [ 6398.597676] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6402.093365] Key type lgssc unregistered [ 6402.359702] LNet: 200351:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6402.384050] LNet: Removed LNI 192.168.202.103@tcp [ 6403.419819] Key type .llcrypt unregistered [ 6403.421933] Key type ._llcrypt unregistered [ 6429.218862] alg: No test for adler32 (adler32-zlib) [ 6429.985278] Key type ._llcrypt registered [ 6429.986412] Key type .llcrypt registered [ 6430.180038] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6431.866942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6432.428477] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 6432.724809] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6432.728834] LNet: Accept secure, port 988 [ 6434.448188] Key type lgssc registered [ 6435.453652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.133120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.713348] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6444.766513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6450.018920] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6450.145767] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.280425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.000686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.379079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6464.932987] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6475.100759] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6482.168878] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6489.885487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6490.337056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.476375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 6495.566295] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6504.006733] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6507.216048] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6511.817471] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6526.602861] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6539.745460] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6539.752714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.761751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6539.781348] Lustre: Skipped 1 previous similar message [ 6539.800983] Lustre: Skipped 1 previous similar message [ 6544.279064] Lustre: server umount lustre-OST0000 complete [ 6549.986087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.005421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.106428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6555.118887] Lustre: Skipped 2 previous similar messages [ 6555.615896] Lustre: server umount lustre-MDT0000 complete [ 6556.134032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.151447] LustreError: Skipped 1 previous similar message [ 6560.113315] LustreError: 204571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194005 with bad export cookie 13235709322808786075 [ 6560.119196] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6560.138076] LustreError: 204571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6560.502744] Lustre: server umount lustre-MDT0001 complete [ 6573.230734] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6576.743435] Key type lgssc unregistered [ 6576.976531] LNet: 205681:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6577.017279] LNet: Removed LNI 192.168.202.103@tcp [ 6577.892909] Key type .llcrypt unregistered [ 6577.894946] Key type ._llcrypt unregistered [ 6605.659529] alg: No test for adler32 (adler32-zlib) [ 6606.413605] Key type ._llcrypt registered [ 6606.416177] Key type .llcrypt registered [ 6606.579670] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6607.988907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6608.672839] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 6608.841375] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6608.843656] LNet: Accept secure, port 988 [ 6610.488451] Key type lgssc registered [ 6611.722628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6620.052910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.680118] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.782702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6626.785899] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6627.313401] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6631.909944] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.799512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6637.033520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.439462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.666201] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6654.402309] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6662.442348] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6671.428734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.963486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.041491] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 6678.577767] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6688.313308] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6694.568695] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6707.536773] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6718.618650] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6725.914243] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6734.821943] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6734.837740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.860361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.915190] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6738.920331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.930288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.035913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.038193] Lustre: Skipped 1 previous similar message [ 6749.154381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6749.156191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6749.163529] Lustre: Skipped 1 previous similar message [ 6749.328781] Lustre: server umount lustre-OST0000 complete [ 6754.275624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.301753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6754.314375] Lustre: Skipped 1 previous similar message [ 6759.477473] Lustre: server umount lustre-MDT0000 complete [ 6764.513391] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.544071] LustreError: Skipped 1 previous similar message [ 6765.142508] LustreError: 207148:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194210 with bad export cookie 16436519454160159443 [ 6765.145741] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6765.161703] LustreError: 207148:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6765.716530] Lustre: server umount lustre-MDT0001 complete [ 6780.808660] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 6783.823544] Key type lgssc unregistered [ 6784.109145] LNet: 212170:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6784.118447] LNet: Removed LNI 192.168.202.103@tcp [ 6784.991579] Key type .llcrypt unregistered [ 6784.998436] Key type ._llcrypt unregistered [ 6811.238855] alg: No test for adler32 (adler32-zlib) [ 6811.998828] Key type ._llcrypt registered [ 6812.001794] Key type .llcrypt registered [ 6812.166636] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 6813.220660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6814.303325] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 6814.649135] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 6814.653570] LNet: Accept secure, port 988 [ 6816.336900] Key type lgssc registered [ 6817.954824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6826.714712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.335136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.423482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.011727] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6833.640394] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.761091] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.859659] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.024898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.638224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6849.191682] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6857.816292] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6864.814801] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6872.672990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.204642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.348554] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 6878.487820] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6888.710176] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6897.746663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.810639] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6897.976125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.064048] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 6912.467694] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6914.553816] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6926.809079] Lustre: Permanently deactivating lustre-OST0001 [ 6926.826359] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6934.639368] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6951.226448] Lustre: Permanently reactivating lustre-OST0001 [ 6951.240985] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6951.246410] Lustre: Skipped 2 previous similar messages [ 6959.671752] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.689814] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6959.709398] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6959.723736] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6959.739343] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34 to 0x2c0000400:65 [ 6959.778078] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6959.820789] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34 to 0x0:65 [ 6960.625442] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.630272] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6960.648434] Lustre: Skipped 2 previous similar messages [ 6962.067092] Lustre: server umount lustre-OST0001 complete [ 6965.741145] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6965.753985] LustreError: Skipped 1 previous similar message [ 6966.753180] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.770129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.799203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.806351] Lustre: Skipped 1 previous similar message [ 6970.856691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.880441] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.892401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.822401] Lustre: server umount lustre-OST0000 complete [ 6976.994112] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6977.001751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.016517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.021105] Lustre: Skipped 1 previous similar message [ 6981.090641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.097284] Lustre: Skipped 1 previous similar message [ 6983.018908] Lustre: server umount lustre-MDT0000 complete [ 6986.214543] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.232551] LustreError: Skipped 3 previous similar messages [ 6987.264112] LustreError: 213635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194432 with bad export cookie 13486633234940771110 [ 6987.275185] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.832266] Lustre: server umount lustre-MDT0001 complete [ 7000.175611] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7003.292964] Key type lgssc unregistered [ 7003.605966] LNet: 218808:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7003.620379] LNet: Removed LNI 192.168.202.103@tcp [ 7004.373988] Key type .llcrypt unregistered [ 7004.380790] Key type ._llcrypt unregistered [ 7040.773635] alg: No test for adler32 (adler32-zlib) [ 7041.528956] Key type ._llcrypt registered [ 7041.531965] Key type .llcrypt registered [ 7041.674524] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7056.148665] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7057.695260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7058.308517] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 7058.638232] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7058.642269] LNet: Accept secure, port 988 [ 7060.328551] Key type lgssc registered [ 7061.319960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7070.001794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7077.461609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.083697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7092.053398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7100.061651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7100.141426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.506112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7101.586492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7101.744686] Lustre: lustre-MDT0000: new disk, initializing [ 7101.825302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7101.843958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7106.683613] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7118.294028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.334974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.411233] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7118.453949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7118.459081] Lustre: Skipped 1 previous similar message [ 7118.566598] Lustre: lustre-MDT0001: new disk, initializing [ 7118.764264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7118.833618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7118.849094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7124.031065] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7134.667332] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7141.862345] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7148.866520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7148.951183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.958600] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7149.331426] Lustre: lustre-OST0000: new disk, initializing [ 7149.336985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7149.472860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.327351] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7155.273306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7155.283235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7163.347782] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7168.096550] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7168.409093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7172.773341] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7172.962106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7176.162049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.163878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.165738] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.176212] Lustre: Skipped 1 previous similar message [ 7176.188787] Lustre: Skipped 1 previous similar message [ 7180.907037] Lustre: server umount lustre-OST0000 complete [ 7185.377459] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7185.388266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.421790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.404896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.440120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.455065] Lustre: Skipped 1 previous similar message [ 7190.612656] Lustre: server umount lustre-MDT0000 complete [ 7191.524897] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.544410] LustreError: Skipped 1 previous similar message [ 7192.560081] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7192.610176] LustreError: Skipped 1 previous similar message [ 7195.285467] LustreError: 225007:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194640 with bad export cookie 1701037743627061316 [ 7195.310446] LustreError: 225007:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7195.328057] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.165117] Lustre: server umount lustre-MDT0001 complete [ 7205.895945] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 15:24:10 (1776194650) [ 7213.154784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.604975] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.750877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7218.036849] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7219.172430] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.288988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7227.880663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.337423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7234.246750] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7244.023651] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7251.144363] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7257.985642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7258.381769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7263.225251] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7271.362571] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.065363] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7281.117580] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7287.871620] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7296.748752] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7313.378562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7313.386848] Lustre: Skipped 1 previous similar message [ 7346.787798] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7346.793840] Lustre: Skipped 4 previous similar messages [ 7415.960941] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 7415.981550] Lustre: Skipped 8 previous similar messages [ 7441.889682] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.920744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.921523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7441.950505] Lustre: Skipped 1 previous similar message [ 7447.010899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7447.018249] Lustre: Skipped 2 previous similar messages [ 7447.465134] Lustre: server umount lustre-OST0000 complete [ 7452.130460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.252623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.270800] Lustre: Skipped 3 previous similar messages [ 7458.065158] Lustre: server umount lustre-MDT0000 complete [ 7462.380112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7462.407769] LustreError: Skipped 1 previous similar message [ 7462.788053] LustreError: 231005:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194908 with bad export cookie 1701037743627062163 [ 7462.791041] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.801844] LustreError: 231005:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7476.779440] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7480.168144] Key type lgssc unregistered [ 7480.770677] LNet: 232337:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7480.780929] LNet: Removed LNI 192.168.202.103@tcp [ 7482.248744] Key type .llcrypt unregistered [ 7482.252159] Key type ._llcrypt unregistered [ 7509.229685] alg: No test for adler32 (adler32-zlib) [ 7509.987704] Key type ._llcrypt registered [ 7509.989636] Key type .llcrypt registered [ 7510.129728] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7511.414419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7511.931373] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 7512.313026] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7512.319274] LNet: Accept secure, port 988 [ 7513.976395] Key type lgssc registered [ 7515.354093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7524.140091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7525.673439] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.726519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.763057] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.364215] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7535.085529] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.193691] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.621311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.329332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7545.700399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.057671] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7559.005146] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7565.004549] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7573.038759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7573.668602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.718502] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug -1 all 8 [ 7588.159760] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7611.882567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.891581] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7611.899827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7611.912395] Lustre: Skipped 1 previous similar message [ 7613.199617] Lustre: server umount lustre-OST0000 complete [ 7622.113642] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7622.117323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.139346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7622.145055] Lustre: Skipped 1 previous similar message [ 7622.151439] Lustre: Skipped 2 previous similar messages [ 7623.781791] Lustre: server umount lustre-MDT0000 complete [ 7627.233698] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7627.269361] LustreError: Skipped 1 previous similar message [ 7628.539113] LustreError: 233800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195073 with bad export cookie 6293942243505102021 [ 7628.544705] LustreError: 166-1: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7628.554026] LustreError: 233800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7628.930807] Lustre: server umount lustre-MDT0001 complete [ 7643.255726] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7646.846688] Key type lgssc unregistered [ 7647.227219] LNet: 237636:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7647.240732] LNet: Removed LNI 192.168.202.103@tcp [ 7647.977888] Key type .llcrypt unregistered [ 7647.979821] Key type ._llcrypt unregistered [ 7673.363597] alg: No test for adler32 (adler32-zlib) [ 7674.131397] Key type ._llcrypt registered [ 7674.139970] Key type .llcrypt registered [ 7674.306444] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7674.666452] Key type .llcrypt unregistered [ 7674.668910] Key type ._llcrypt unregistered [ 7704.616626] alg: No test for adler32 (adler32-zlib) [ 7705.377853] Key type ._llcrypt registered [ 7705.379540] Key type .llcrypt registered [ 7705.504708] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_hostid [ 7720.934851] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local [ 7722.316124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7723.025897] Lustre: Lustre: Build Version: 2.15.8_12_gc5b382e [ 7723.259201] LNet: Added LNI 192.168.202.103@tcp [8/256/0/180] [ 7723.262907] LNet: Accept secure, port 988 [ 7724.904365] Key type lgssc registered [ 7726.436469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7740.432422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7749.128285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7756.573112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7764.858211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7766.813515] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7469 sec ========== 15:33:30 (1776195210) [ 7817.374509] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing unload_modules_local [ 7820.554732] Key type lgssc unregistered [ 7820.910837] LNet: 243907:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.919359] LNet: Removed LNI 192.168.202.103@tcp [ 7821.716762] Key type .llcrypt unregistered [ 7821.719680] Key type ._llcrypt unregistered