[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481978111 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002313] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007300] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011039] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.021695] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026582] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.030427] *** VALIDATE proc *** [ 0.031204] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040195] debug: unmapping init [mem 0xffffffffbd859000-0xffffffffbd860fff] [ 0.042214] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051296] rcu: Hierarchical SRCU implementation. [ 0.053494] smp: Bringing up secondary CPUs ... [ 0.054654] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.067146] smp: Brought up 1 node, 4 CPUs [ 0.069015] smpboot: Max logical packages: 1 [ 0.070010] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.099245] node 0 deferred pages initialised in 27ms [ 0.103169] devtmpfs: initialized [ 0.104323] x86/mm: Memory block size: 128MB [ 0.107226] gcov: version magic: 0x41383552 [ 0.109122] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.110216] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113478] pinctrl core: initialized pinctrl subsystem [ 0.115190] [ 0.115637] ************************************************************* [ 0.118013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120012] ** ** [ 0.123012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126013] ** ** [ 0.127009] ** This means that this kernel is built to expose internal ** [ 0.129011] ** IOMMU data structures, which may compromise security on ** [ 0.131011] ** your system. ** [ 0.135013] ** ** [ 0.137008] ** If you see this message and you are not debugging the ** [ 0.138007] ** kernel, report this immediately to your vendor! ** [ 0.140015] ** ** [ 0.143011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145011] ************************************************************* [ 0.147500] NET: Registered protocol family 16 [ 0.149389] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157017] cpuidle: using governor menu [ 0.158679] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162621] PCI: Using configuration type 1 for base access [ 0.164168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173184] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182030] cryptd: max_cpu_qlen set to 1000 [ 0.183253] ACPI: Added _OSI(Module Device) [ 0.184000] ACPI: Added _OSI(Processor Device) [ 0.184000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192160] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200072] ACPI: Interpreter enabled [ 0.201052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203010] ACPI: Using IOAPIC for interrupt routing [ 0.204189] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208651] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223812] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233188] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238790] acpiphp: Slot [2] registered [ 0.240210] acpiphp: Slot [5] registered [ 0.242095] acpiphp: Slot [6] registered [ 0.243097] acpiphp: Slot [7] registered [ 0.245119] acpiphp: Slot [8] registered [ 0.247117] acpiphp: Slot [9] registered [ 0.249133] acpiphp: Slot [10] registered [ 0.250119] acpiphp: Slot [3] registered [ 0.252164] acpiphp: Slot [4] registered [ 0.255712] acpiphp: Slot [11] registered [ 0.258115] acpiphp: Slot [12] registered [ 0.260079] acpiphp: Slot [13] registered [ 0.261224] acpiphp: Slot [14] registered [ 0.263109] acpiphp: Slot [15] registered [ 0.265113] acpiphp: Slot [16] registered [ 0.267159] acpiphp: Slot [17] registered [ 0.268195] acpiphp: Slot [18] registered [ 0.270066] acpiphp: Slot [19] registered [ 0.271063] acpiphp: Slot [20] registered [ 0.272133] acpiphp: Slot [21] registered [ 0.273000] acpiphp: Slot [22] registered [ 0.273000] acpiphp: Slot [23] registered [ 0.275128] acpiphp: Slot [24] registered [ 0.277148] acpiphp: Slot [25] registered [ 0.279156] acpiphp: Slot [26] registered [ 0.281110] acpiphp: Slot [27] registered [ 0.283094] acpiphp: Slot [28] registered [ 0.284153] acpiphp: Slot [29] registered [ 0.286094] acpiphp: Slot [30] registered [ 0.288638] acpiphp: Slot [31] registered [ 0.290048] PCI host bridge to bus 0000:00 [ 0.292012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309152] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315530] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329644] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.335021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340468] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342395] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346804] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.366940] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.370932] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.377706] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403828] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.410013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.418019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.424000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.430014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.433015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.442015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.449186] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.456013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.461015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.473000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.480000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.487011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.491012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.502015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.507923] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.522011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.537018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.546305] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.548355] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.551400] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.552376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.556227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.562174] iommu: Default domain type: Passthrough [ 0.573432] SCSI subsystem initialized [ 0.621159] ACPI: bus type USB registered [ 0.623130] usbcore: registered new interface driver usbfs [ 0.625042] usbcore: registered new interface driver hub [ 0.626060] usbcore: registered new device driver usb [ 0.628101] pps_core: LinuxPPS API ver. 1 registered [ 0.630009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.632041] PTP clock support registered [ 0.637052] EDAC MC: Ver: 3.0.0 [ 0.638518] PCI: Using ACPI for IRQ routing [ 0.639952] NetLabel: Initializing [ 0.641008] NetLabel: domain hash size = 128 [ 0.643016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.645164] NetLabel: unlabeled traffic allowed by default [ 0.648412] vgaarb: loaded [ 0.650616] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.652046] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.661429] clocksource: Switched to clocksource kvm-clock [ 0.777437] VFS: Disk quotas dquot_6.6.0 [ 0.779023] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.781736] *** VALIDATE ramfs *** [ 0.783081] *** VALIDATE hugetlbfs *** [ 0.785376] pnp: PnP ACPI init [ 0.787918] pnp: PnP ACPI: found 6 devices [ 0.803771] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.807261] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.808912] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.810942] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.813411] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.815854] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.818590] NET: Registered protocol family 2 [ 0.821256] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.826862] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.830836] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.836919] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.840626] TCP: Hash tables configured (established 65536 bind 65536) [ 0.844034] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.846927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849402] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852987] NET: Registered protocol family 1 [ 0.857784] RPC: Registered named UNIX socket transport module. [ 0.860576] RPC: Registered udp transport module. [ 0.862550] RPC: Registered tcp transport module. [ 0.864404] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.867160] NET: Registered protocol family 44 [ 0.869107] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.871668] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.873927] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.876426] PCI: CLS 0 bytes, default 64 [ 0.878376] Unpacking initramfs... [ 3.126637] debug: unmapping init [mem 0xffff8c6ffcc54000-0xffff8c6ffffbffff] [ 3.130807] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.132712] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.134893] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.828723] Initialise system trusted keyrings [ 3.830097] Key type blacklist registered [ 3.833719] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.846151] zbud: loaded [ 3.851256] *** VALIDATE nfs *** [ 3.853303] *** VALIDATE nfs4 *** [ 3.854947] pstore: using deflate compression [ 3.860406] Platform Keyring initialized [ 4.038722] NET: Registered protocol family 38 [ 4.041107] Key type asymmetric registered [ 4.042297] Asymmetric key parser 'x509' registered [ 4.046782] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.050126] io scheduler mq-deadline registered [ 4.051377] io scheduler kyber registered [ 4.053214] io scheduler bfq registered [ 4.054814] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.058772] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.063952] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.067464] ACPI: Power Button [PWRF] [ 4.084600] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.091247] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.110409] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.118305] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.139697] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.170633] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.201546] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.205726] Non-volatile memory driver v1.3 [ 4.207248] Linux agpgart interface v0.103 [ 4.234583] virtio_blk virtio1: [vda] 68264 512-byte logical blocks (35.0 MB/33.3 MiB) [ 4.236876] vda: detected capacity change from 0 to 34951168 [ 4.250593] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.253844] vdb: detected capacity change from 0 to 1073741824 [ 4.268628] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.271552] vdc: detected capacity change from 0 to 2621440000 [ 4.285623] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.288931] vdd: detected capacity change from 0 to 2621440000 [ 4.305722] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.309898] vde: detected capacity change from 0 to 4294967296 [ 4.327497] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.330676] vdf: detected capacity change from 0 to 4294967296 [ 4.341325] libphy: Fixed MDIO Bus: probed [ 4.357941] usbcore: registered new interface driver usbserial_generic [ 4.361394] usbserial: USB Serial support registered for generic [ 4.364403] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.369724] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.371707] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.374097] mousedev: PS/2 mouse device common for all mice [ 4.377941] rtc_cmos 00:05: RTC can wake from S4 [ 4.381282] rtc_cmos 00:05: registered as rtc0 [ 4.383576] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.385597] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.391966] intel_pstate: CPU model not supported [ 4.395151] hid: raw HID events driver (C) Jiri Kosina [ 4.397522] usbcore: registered new interface driver usbhid [ 4.399980] usbhid: USB HID core driver [ 4.401955] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.405365] drop_monitor: Initializing network drop monitor service [ 4.407759] Initializing XFRM netlink socket [ 4.410098] NET: Registered protocol family 10 [ 4.412701] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.417728] Segment Routing with IPv6 [ 4.419146] NET: Registered protocol family 17 [ 4.421734] mpls_gso: MPLS GSO support [ 4.428855] RAS: Correctable Errors collector initialized. [ 4.432218] AVX version of gcm_enc/dec engaged. [ 4.433918] AES CTR mode by8 optimization enabled [ 4.570737] sched_clock: Marking stable (4570587591, 0)->(5559290552, -988702961) [ 4.577677] registered taskstats version 1 [ 4.580197] Loading compiled-in X.509 certificates [ 4.582574] zswap: loaded using pool lzo/zbud [ 4.615191] Key type big_key registered [ 4.633121] Key type encrypted registered [ 4.635142] ima: No TPM chip found, activating TPM-bypass! [ 4.643359] ima: Allocated hash algorithm: sha1 [ 4.647066] ima: No architecture policies found [ 4.671979] evm: Initialising EVM extended attributes: [ 4.674267] evm: security.selinux [ 4.677818] evm: security.ima [ 4.679078] evm: security.capability [ 4.680799] evm: HMAC attrs: 0x1 [ 4.683796] rtc_cmos 00:05: setting system clock to 2026-06-09 23:05:28 UTC (1781046328) [ 4.692866] debug: unmapping init [mem 0xffffffffbe803000-0xffffffffbe9fffff] [ 4.697813] debug: unmapping init [mem 0xffffffffbd582000-0xffffffffbd858fff] [ 4.707129] Write protecting the kernel read-only data: 28672k [ 4.710669] debug: unmapping init [mem 0xffffffffbbc03000-0xffffffffbbdfffff] [ 4.713696] debug: unmapping init [mem 0xffffffffbc514000-0xffffffffbc5fffff] [ 4.756769] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.766350] systemd[1]: Detected virtualization kvm. [ 4.768730] systemd[1]: Detected architecture x86-64. [ 4.771111] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.796786] systemd[1]: No hostname configured. [ 4.798546] systemd[1]: Set hostname to . [ 4.800756] random: systemd: uninitialized urandom read (16 bytes read) [ 4.803205] systemd[1]: Initializing machine ID from random generator. [ 4.903139] random: ln: uninitialized urandom read (6 bytes read) [ 5.090483] random: systemd: uninitialized urandom read (16 bytes read) [ 5.095131] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.104649] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.123273] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.162244] device-mapper: uevent: version 1.0.3 [ 6.164644] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.416786] virtio_net virtio0 ens2: renamed from eth0 [ 7.463683] random: fast init done [ 7.627159] scsi host0: ata_piix [ 7.697823] scsi host1: ata_piix [ 7.699791] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.702189] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.186766] random: crng init done [ 13.188117] random: 7 urandom warning(s) missed due to ratelimiting [ 13.752174] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.584880] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.333232] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.833596] SELinux: Disabled at runtime. [ 17.932698] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.955815] systemd[1]: Detected virtualization kvm. [ 17.959195] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.289314] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.293652] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.306093] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.311162] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.318029] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.346172] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.383612] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ 19.586938] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.475395] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.220577] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.475675] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.898929] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.947731] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 26.448908] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 26.996743] NFS: Registering the id_resolver key type [ 26.999321] Key type id_resolver registered [ 27.001411] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg227-server login: [ 50.370595] libcfs: loading out-of-tree module taints kernel. [ 50.383650] alg: No test for adler32 (adler32-zlib) [ 51.135410] Key type ._llcrypt registered [ 51.136764] Key type .llcrypt registered [ 51.185903] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 58.280380] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 58.879820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.104563] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 59.380855] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 59.384406] LNet: Accept secure, port 988 [ 60.991116] Key type lgssc registered [ 61.499759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.988932] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.174783] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.064894] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.962895] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.862883] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.398775] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 88.957095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.985560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.073437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 90.085245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 90.125777] Lustre: lustre-MDT0000: new disk, initializing [ 90.154867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.162658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 91.813639] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 97.382888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.407113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.430658] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 97.439144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 97.440812] Lustre: Skipped 1 previous similar message [ 97.478891] Lustre: lustre-MDT0001: new disk, initializing [ 97.499060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 97.508879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 97.511919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 98.904272] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 104.224125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.253643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 104.256576] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 104.338387] Lustre: lustre-OST0000: new disk, initializing [ 104.340450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 104.359169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.752234] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 111.090768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.126726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 111.130102] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 111.165537] Lustre: lustre-OST0001: new disk, initializing [ 111.169870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 111.189553] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.798904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 111.802520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 111.807427] Lustre: Skipped 1 previous similar message [ 112.735301] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 118.924530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.893618] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 130.261565] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.949670] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing yml_node [ 133.513879] Lustre: DEBUG MARKER: Client: 2.15.8.5 [ 134.466781] Lustre: DEBUG MARKER: MDS: 2.15.8.5 [ 135.489080] Lustre: DEBUG MARKER: OSS: 2.15.8.5 [ 136.088984] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 9 19:07:38 EDT 2026 [ 139.220800] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 139.748794] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 153.057411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.059523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 153.062972] Lustre: Skipped 2 previous similar messages [ 153.065775] Lustre: Skipped 1 previous similar message [ 157.666528] Lustre: server umount lustre-MDT0000 complete [ 158.176192] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 158.185859] LustreError: Skipped 3 previous similar messages [ 159.516215] LustreError: 8230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046483 with bad export cookie 15413022046448437322 [ 159.516683] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 159.520018] LustreError: 8230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 163.295614] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 163.295986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.296237] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 163.296241] Lustre: Skipped 2 previous similar messages [ 163.300595] LustreError: Skipped 1 previous similar message [ 163.304692] Lustre: Skipped 2 previous similar messages [ 164.319705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 164.319781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 164.323324] Lustre: Skipped 2 previous similar messages [ 164.328752] LustreError: Skipped 1 previous similar message [ 165.703546] Lustre: server umount lustre-MDT0001 complete [ 167.325327] Lustre: server umount lustre-OST0000 complete [ 171.416959] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 174.648963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 178.353644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.654105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.790311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.037804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.838078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.878905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.985880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 188.000506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 188.046286] Lustre: lustre-MDT0000: new disk, initializing [ 188.101932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.109518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.112486] Lustre: Skipped 1 previous similar message [ 189.761884] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 194.677808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.707081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.753395] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 194.870744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.876647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.670370] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 201.539539] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 203.876940] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 206.631663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.663179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.666891] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 206.779803] Lustre: lustre-OST0000: new disk, initializing [ 206.781167] Lustre: Skipped 1 previous similar message [ 206.784698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 206.787769] Lustre: Skipped 2 previous similar messages [ 206.818419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.823377] Lustre: Skipped 1 previous similar message [ 208.566209] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 208.697232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.700931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.751561] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 214.490281] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 214.583201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 216.080229] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 216.158673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 219.105029] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 219.105377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.106241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 219.106244] Lustre: Skipped 2 previous similar messages [ 219.107867] LustreError: Skipped 1 previous similar message [ 219.112908] Lustre: Skipped 1 previous similar message [ 224.226942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 224.229597] Lustre: Skipped 2 previous similar messages [ 231.391110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 231.455686] Lustre: server umount lustre-OST0000 complete [ 231.456978] Lustre: Skipped 1 previous similar message [ 234.464320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.464866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 234.468421] Lustre: Skipped 1 previous similar message [ 234.470190] Lustre: Skipped 2 previous similar messages [ 247.263104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 247.308887] Lustre: server umount lustre-MDT0000 complete [ 248.638580] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046572 with bad export cookie 15413022046448438680 [ 248.641463] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.642595] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 250.010358] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:09:32 (1781046572) [ 252.263719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.416389] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.421787] LustreError: Skipped 2 previous similar messages [ 252.441276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.934389] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 256.770481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.306437] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 260.977748] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 263.145460] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 265.719559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.340021] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 270.166184] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 277.474185] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 277.474955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.477515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 277.477521] Lustre: Skipped 5 previous similar messages [ 277.484909] Lustre: Skipped 1 previous similar message [ 282.908908] Lustre: server umount lustre-OST0000 complete [ 282.911370] Lustre: Skipped 1 previous similar message [ 287.712957] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.712958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.799449] LustreError: 18693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046615 with bad export cookie 15413022046448439534 [ 291.799661] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 291.804646] LustreError: 18693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 297.632317] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 298.962155] Key type lgssc unregistered [ 299.130675] LNet: 22365:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.134764] LNet: Removed LNI 192.168.202.127@tcp [ 299.534198] Key type .llcrypt unregistered [ 299.535831] Key type ._llcrypt unregistered [ 310.042874] alg: No test for adler32 (adler32-zlib) [ 310.795073] Key type ._llcrypt registered [ 310.798119] Key type .llcrypt registered [ 310.851775] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 311.318035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.625637] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 311.732817] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 311.735100] LNet: Accept secure, port 988 [ 313.343126] Key type lgssc registered [ 313.858373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 316.958021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.149308] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.178196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 319.918404] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 323.449779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.584752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 325.118312] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 327.832268] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 330.086853] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 332.582315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 332.706446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.734847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 334.215031] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 337.036574] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 344.034034] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 344.035306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.037353] LustreError: Skipped 1 previous similar message [ 344.042954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 344.047105] Lustre: Skipped 1 previous similar message [ 348.119504] Lustre: server umount lustre-OST0000 complete [ 354.274821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.275070] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.275572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.279621] Lustre: Skipped 2 previous similar messages [ 355.598721] Lustre: server umount lustre-MDT0000 complete [ 357.133417] LustreError: 23806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046680 with bad export cookie 699182791778159736 [ 357.133820] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.143773] LustreError: 23806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 357.285123] Lustre: server umount lustre-MDT0001 complete [ 362.360095] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 363.510245] Key type lgssc unregistered [ 363.641423] LNet: 27827:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.646644] LNet: Removed LNI 192.168.202.127@tcp [ 363.969431] Key type .llcrypt unregistered [ 363.971521] Key type ._llcrypt unregistered [ 373.542248] alg: No test for adler32 (adler32-zlib) [ 374.294365] Key type ._llcrypt registered [ 374.296940] Key type .llcrypt registered [ 374.336747] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 374.679832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.941970] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 375.028519] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 375.030912] LNet: Accept secure, port 988 [ 376.615132] Key type lgssc registered [ 376.984977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 379.927893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.108715] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.138558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.882970] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 386.279470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.437361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.080543] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 390.916835] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 393.024329] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 398.172717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 398.308752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.917417] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 402.591503] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 411.623874] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 416.738625] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 416.738813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.740777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 416.742129] LustreError: Skipped 1 previous similar message [ 416.746966] Lustre: Skipped 1 previous similar message [ 420.727923] Lustre: server umount lustre-OST0000 complete [ 422.368419] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 422.373079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.379993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.382625] Lustre: Skipped 1 previous similar message [ 426.977210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.977685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.420661] Lustre: server umount lustre-MDT0000 complete [ 429.842052] LustreError: 29280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046753 with bad export cookie 4614762649418220698 [ 429.845061] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.846061] LustreError: 29280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 429.964490] Lustre: server umount lustre-MDT0001 complete [ 435.342237] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 436.609977] Key type lgssc unregistered [ 436.725504] LNet: 33318:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.729334] LNet: Removed LNI 192.168.202.127@tcp [ 437.068850] Key type .llcrypt unregistered [ 437.070922] Key type ._llcrypt unregistered [ 447.128368] alg: No test for adler32 (adler32-zlib) [ 447.880359] Key type ._llcrypt registered [ 447.881739] Key type .llcrypt registered [ 447.926376] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 448.465786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 448.610562] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 448.716199] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 448.718413] LNet: Accept secure, port 988 [ 450.311555] Key type lgssc registered [ 450.768046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 454.101225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.282239] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.306924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.979685] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 460.261381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.387893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.949469] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 464.912218] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 467.216618] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 469.764369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.897093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.950266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 471.453219] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 474.141050] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.857836] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 485.861995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.866857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.369378] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 486.370153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.372252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.492932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.497185] Lustre: Skipped 2 previous similar messages [ 496.095375] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 496.158709] Lustre: server umount lustre-OST0000 complete [ 501.215798] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 501.219044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.223989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.915429] Lustre: server umount lustre-MDT0000 complete [ 505.376869] LustreError: 34758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046829 with bad export cookie 3728595292704289418 [ 505.378690] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.381877] LustreError: 34758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 505.514236] Lustre: server umount lustre-MDT0001 complete [ 511.022222] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 512.152628] Key type lgssc unregistered [ 512.265511] LNet: 38432:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.268798] LNet: Removed LNI 192.168.202.127@tcp [ 512.589761] Key type .llcrypt unregistered [ 512.591251] Key type ._llcrypt unregistered [ 521.527956] alg: No test for adler32 (adler32-zlib) [ 522.279403] Key type ._llcrypt registered [ 522.280980] Key type .llcrypt registered [ 522.319987] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 522.715429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.855522] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 522.936045] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 522.938114] LNet: Accept secure, port 988 [ 524.519100] Key type lgssc registered [ 524.854232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 527.563311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.698456] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.713514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.097834] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 532.878347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.986205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 534.314372] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 536.763952] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 538.732776] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 541.169518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.312942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.806226] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 543.402932] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 545.638378] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 553.951828] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.954640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.962046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.310626] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 558.049516] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.053832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.063691] Lustre: server umount lustre-OST0000 complete [ 573.408260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.414514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.417180] Lustre: Skipped 3 previous similar messages [ 578.437778] Lustre: server umount lustre-MDT0000 complete [ 578.530654] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.535019] LustreError: Skipped 1 previous similar message [ 579.921039] LustreError: 42662:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046903 with bad export cookie 2046894399738923290 [ 579.924128] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.925338] LustreError: 42662:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 594.399727] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 594.495555] Lustre: server umount lustre-MDT0001 complete [ 599.310733] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 600.352458] Key type lgssc unregistered [ 600.466453] LNet: 43719:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.469828] LNet: Removed LNI 192.168.202.127@tcp [ 600.765204] Key type .llcrypt unregistered [ 600.767292] Key type ._llcrypt unregistered [ 609.604149] alg: No test for adler32 (adler32-zlib) [ 610.355258] Key type ._llcrypt registered [ 610.356307] Key type .llcrypt registered [ 610.393438] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 610.764959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 610.967091] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 611.042864] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 611.044604] LNet: Accept secure, port 988 [ 612.623119] Key type lgssc registered [ 612.967850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.700974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.857979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.877481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.236853] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 620.996577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.123444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 622.562981] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 625.012886] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 626.760933] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 628.984145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 629.100598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.546298] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 632.164302] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 633.056795] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 641.505895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 641.506924] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.508069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.521099] Lustre: Skipped 2 previous similar messages [ 644.353759] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 644.358451] Lustre: Skipped 2 previous similar messages [ 646.085936] Lustre: server umount lustre-MDT0000 complete [ 646.625062] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.629514] LustreError: Skipped 2 previous similar messages [ 647.437437] LustreError: 45159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781046971 with bad export cookie 9731329291039526967 [ 647.439804] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.441899] LustreError: 45159:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 649.461461] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.467940] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 651.743802] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.743955] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.754373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 653.623161] Lustre: server umount lustre-MDT0001 complete [ 671.642461] Lustre: server umount lustre-OST0000 complete [ 678.521590] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 679.583229] Key type lgssc unregistered [ 679.701480] LNet: 49157:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.706040] LNet: Removed LNI 192.168.202.127@tcp [ 680.031418] Key type .llcrypt unregistered [ 680.033468] Key type ._llcrypt unregistered [ 690.122866] alg: No test for adler32 (adler32-zlib) [ 690.874418] Key type ._llcrypt registered [ 690.875745] Key type .llcrypt registered [ 690.918468] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 691.265222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 691.504650] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 691.590140] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 691.592407] LNet: Accept secure, port 988 [ 693.175104] Key type lgssc registered [ 693.567195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.398237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.887303] LustreError: 50590:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 744.607203] LustreError: 50590:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 774.303698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 776.106554] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 779.450596] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 817.504697] Lustre: server umount lustre-OST0000 complete [ 824.450086] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 825.447645] Key type lgssc unregistered [ 825.569747] LNet: 52257:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.572511] LNet: Removed LNI 192.168.202.127@tcp [ 825.896933] Key type .llcrypt unregistered [ 825.898648] Key type ._llcrypt unregistered [ 835.225775] alg: No test for adler32 (adler32-zlib) [ 835.977473] Key type ._llcrypt registered [ 835.978685] Key type .llcrypt registered [ 836.021708] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 836.480038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.617883] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 836.702580] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 836.705401] LNet: Accept secure, port 988 [ 838.287106] Key type lgssc registered [ 838.647731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.500904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.663874] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.682967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.172088] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 847.290859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.418802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 848.887677] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 851.462314] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.500130] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 855.791664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 855.900241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.243489] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 858.984528] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 859.644234] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 863.073820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.075043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.510067] Lustre: server umount lustre-OST0000 complete [ 867.807824] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 867.810933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.815494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.818022] Lustre: Skipped 1 previous similar message [ 873.440276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.442689] Lustre: Skipped 3 previous similar messages [ 873.800932] Lustre: server umount lustre-MDT0000 complete [ 875.181044] LustreError: 53699:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047198 with bad export cookie 15504771008992372144 [ 875.183180] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 875.185178] LustreError: 53699:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 875.307674] Lustre: server umount lustre-MDT0001 complete [ 880.103893] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 881.127445] Key type lgssc unregistered [ 881.236423] LNet: 57367:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.239137] LNet: Removed LNI 192.168.202.127@tcp [ 881.539968] Key type .llcrypt unregistered [ 881.541951] Key type ._llcrypt unregistered [ 890.535535] alg: No test for adler32 (adler32-zlib) [ 891.286365] Key type ._llcrypt registered [ 891.287623] Key type .llcrypt registered [ 891.323290] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 891.745305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.876667] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 891.950951] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 891.952786] LNet: Accept secure, port 988 [ 893.535123] Key type lgssc registered [ 893.888455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.635557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.120211] LustreError: 58801:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 944.863199] LustreError: 58801:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 975.583182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.068146] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 979.648719] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 981.983199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.152218] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.189326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.212369] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 998.682777] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1001.644556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.762322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1003.176517] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1005.685799] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1007.589367] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.195334] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.198766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.205033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.207232] Lustre: Skipped 1 previous similar message [ 1014.361056] Lustre: server umount lustre-OST0000 complete [ 1028.560484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.561578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.563954] Lustre: Skipped 2 previous similar messages [ 1033.696657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.699206] Lustre: Skipped 2 previous similar messages [ 1034.631557] Lustre: server umount lustre-MDT0000 complete [ 1035.925945] LustreError: 58807:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047359 with bad export cookie 11438251556353642750 [ 1035.928855] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.929690] LustreError: 58807:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1042.108770] Lustre: server umount lustre-MDT0001 complete [ 1046.963617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1047.940182] Key type lgssc unregistered [ 1048.049484] LNet: 62668:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.052811] LNet: Removed LNI 192.168.202.127@tcp [ 1048.345417] Key type .llcrypt unregistered [ 1048.346504] Key type ._llcrypt unregistered [ 1057.074897] alg: No test for adler32 (adler32-zlib) [ 1057.826356] Key type ._llcrypt registered [ 1057.827411] Key type .llcrypt registered [ 1057.864459] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1058.195310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.413254] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1058.488091] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1058.490129] LNet: Accept secure, port 988 [ 1060.071097] Key type lgssc registered [ 1060.408481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.940493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.081112] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.094779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.469094] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1068.207645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.336243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.772255] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1072.192556] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1073.963138] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.123979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.229357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.619987] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1079.332584] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1080.122256] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1093.604369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.605222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.606429] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.608549] Lustre: Skipped 1 previous similar message [ 1098.722292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.724724] Lustre: Skipped 2 previous similar messages [ 1098.774732] Lustre: server umount lustre-OST0000 complete [ 1103.840502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.840933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.846168] Lustre: Skipped 1 previous similar message [ 1103.849952] Lustre: Skipped 1 previous similar message [ 1106.119950] Lustre: server umount lustre-MDT0000 complete [ 1107.458615] LustreError: 64108:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047431 with bad export cookie 1313674227786191741 [ 1107.459359] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.462501] LustreError: 64108:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1107.582039] Lustre: server umount lustre-MDT0001 complete [ 1112.293316] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1113.293114] Key type lgssc unregistered [ 1113.402346] LNet: 67809:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.405306] LNet: Removed LNI 192.168.202.127@tcp [ 1113.712890] Key type .llcrypt unregistered [ 1113.714043] Key type ._llcrypt unregistered [ 1125.451631] alg: No test for adler32 (adler32-zlib) [ 1126.203366] Key type ._llcrypt registered [ 1126.204457] Key type .llcrypt registered [ 1126.238540] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1126.545372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.782567] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1126.859014] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1126.860936] LNet: Accept secure, port 988 [ 1128.439083] Key type lgssc registered [ 1128.775794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.334948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.480663] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.496113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.923976] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1136.790374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.902525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1138.409125] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1141.010543] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.986795] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1145.199682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1145.297945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.682417] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1147.365247] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 1149.187549] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1156.463255] Lustre: Failing over lustre-MDT0000 [ 1156.577839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.579309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.583070] Lustre: Skipped 2 previous similar messages [ 1156.585285] Lustre: Skipped 1 previous similar message [ 1156.590135] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.597909] Lustre: server umount lustre-MDT0000 complete [ 1159.609528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.648912] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.648977] LustreError: 11-0: MGC192.168.202.127@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1159.649361] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.650421] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0xf247c9e329e39a14 to 0xf247c9e329e39de8 [ 1159.652745] LustreError: Skipped 2 previous similar messages [ 1159.664314] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 1159.703257] Lustre: *** cfs_fail_loc=135, val=0*** [ 1159.743946] BUG: spinlock bad magic on CPU#1, llog_process_th/72716 [ 1159.745891] lock: 0xffff8c706f6640c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1159.747833] CPU: 1 PID: 72716 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1159.750529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 1159.752253] Call Trace: [ 1159.753711] ? dump_stack+0xbb/0x10e [ 1159.754444] ? spin_dump.cold.3+0x32/0x65 [ 1159.755436] ? do_raw_spin_lock+0xce/0x190 [ 1159.756810] ? _raw_spin_lock+0x21/0x30 [ 1159.758129] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1159.759835] ? mdd_process_config+0x368/0x9c0 [mdd] [ 1159.761227] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 1159.762400] ? mdt_init0+0x650/0x1750 [mdt] [ 1159.763400] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1159.764349] ? obd_setup+0x196/0x3e0 [obdclass] [ 1159.765292] ? class_setup+0x6ee/0x9e0 [obdclass] [ 1159.766229] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 1159.767324] ? class_config_llog_handler+0x8c1/0x1ab0 [obdclass] [ 1159.768619] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 1159.770147] ? llog_process_thread+0x1506/0x2410 [obdclass] [ 1159.771708] ? mgs_key_init+0xdd/0x110 [mgs] [ 1159.772669] ? keys_fill+0x10d/0x1b0 [obdclass] [ 1159.773581] ? lu_context_init+0xc8/0x240 [obdclass] [ 1159.774537] ? llog_process+0x40/0x40 [obdclass] [ 1159.775500] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 1159.776877] ? kthread+0x1d1/0x200 [ 1159.777798] ? set_kthread_struct+0x70/0x70 [ 1159.778633] ? ret_from_fork+0x1f/0x30 [ 1159.780581] LustreError: 72716:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 1159.782557] LustreError: 72716:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.127@tcp: cfg command failed: rc = -2 [ 1159.785045] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1159.785045] [ 1159.787568] LustreError: 15c-8: MGC192.168.202.127@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1159.791743] LustreError: 72704:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1159.794381] LustreError: 72704:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1159.797277] LustreError: 72704:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1159.804949] Lustre: server umount lustre-MDT0000 complete [ 1159.806116] LustreError: 72704:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1161.904454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.947863] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.947928] LustreError: 11-0: MGC192.168.202.127@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1161.948222] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.948228] LustreError: Skipped 1 previous similar message [ 1161.951567] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0xf247c9e329e39de8 to 0xf247c9e329e39e6d [ 1161.961531] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 1162.033158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.063649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1163.322384] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1166.069305] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1167.334710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1167.340317] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1167.354661] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 1171.935700] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.938161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.941717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.349086] Lustre: server umount lustre-OST0000 complete [ 1177.568537] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1177.568611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.568951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.568955] Lustre: Skipped 2 previous similar messages [ 1177.571158] LustreError: Skipped 1 previous similar message [ 1177.575384] Lustre: Skipped 2 previous similar messages [ 1181.639885] Lustre: server umount lustre-MDT0000 complete [ 1182.688070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.693142] LustreError: Skipped 3 previous similar messages [ 1182.847912] LustreError: 71508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047506 with bad export cookie 17458144458022559341 [ 1182.850575] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.852458] LustreError: 71508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1187.209144] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1188.157941] Key type lgssc unregistered [ 1188.265290] LNet: 74597:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1188.267438] LNet: Removed LNI 192.168.202.127@tcp [ 1188.547437] Key type .llcrypt unregistered [ 1188.548331] Key type ._llcrypt unregistered [ 1196.461177] alg: No test for adler32 (adler32-zlib) [ 1197.212348] Key type ._llcrypt registered [ 1197.213181] Key type .llcrypt registered [ 1197.248051] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1197.569889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.750418] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1197.821472] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1197.823433] LNet: Accept secure, port 988 [ 1199.399077] Key type lgssc registered [ 1199.686613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.096742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.220754] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.233300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.531055] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1207.234312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.352321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1208.617605] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1210.907427] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1212.637283] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1214.713925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.818703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.149670] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1218.525993] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1218.915606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 1226.909254] Lustre: Failing over lustre-MDT0000 [ 1227.050839] Lustre: server umount lustre-MDT0000 complete [ 1227.743904] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.747166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.752588] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.257536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.261079] Lustre: Skipped 1 previous similar message [ 1230.074402] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.077560] LustreError: Skipped 3 previous similar messages [ 1230.189140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.222542] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.226171] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0x91c9c5cac90f0a53 to 0x91c9c5cac90f0e2e [ 1230.229335] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 1230.266727] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1230.309028] LustreError: 79212:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 1230.311485] LustreError: 79212:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.127@tcp: cfg command failed: rc = -9 [ 1230.314533] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1230.314533] [ 1230.317167] LustreError: 15c-8: MGC192.168.202.127@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1230.320734] LustreError: 79200:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1230.323114] LustreError: 79200:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 1230.325271] LustreError: 79200:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1230.333362] Lustre: server umount lustre-MDT0000 complete [ 1230.334672] LustreError: 79200:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1232.318342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.350028] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.350946] LustreError: 11-0: MGC192.168.202.127@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1232.350975] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.352164] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0x91c9c5cac90f0e2e to 0x91c9c5cac90f0eba [ 1232.354126] LustreError: Skipped 5 previous similar messages [ 1232.361232] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 1232.434834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.461426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1233.746332] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1235.188980] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1237.476515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1237.480799] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1237.493144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 1238.495624] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1238.497898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.502056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.592759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1242.593866] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1242.597897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.439810] Lustre: server umount lustre-OST0000 complete [ 1247.712516] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1247.712773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.715338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.720376] Lustre: Skipped 1 previous similar message [ 1247.724169] Lustre: Skipped 2 previous similar messages [ 1252.832112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.834365] Lustre: Skipped 1 previous similar message [ 1257.951911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.953971] Lustre: Skipped 1 previous similar message [ 1259.999089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1260.037418] Lustre: server umount lustre-MDT0000 complete [ 1261.290906] LustreError: 78703:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047585 with bad export cookie 10505145080565665466 [ 1261.293740] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.296131] LustreError: 78703:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1265.954248] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1266.936680] Key type lgssc unregistered [ 1267.051344] LNet: 81094:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.054641] LNet: Removed LNI 192.168.202.127@tcp [ 1267.365634] Key type .llcrypt unregistered [ 1267.367077] Key type ._llcrypt unregistered [ 1275.431725] alg: No test for adler32 (adler32-zlib) [ 1276.183362] Key type ._llcrypt registered [ 1276.184561] Key type .llcrypt registered [ 1276.218254] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1276.616093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.760746] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1276.841347] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1276.843192] LNet: Accept secure, port 988 [ 1278.423061] Key type lgssc registered [ 1278.698970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1280.982226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.098726] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.109863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.370180] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1285.845550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.941096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.158486] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1289.364197] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.157232] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1293.173142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1293.277714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.533832] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1296.357583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 1296.793615] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.568942] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.569404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.571643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.573750] Lustre: Skipped 1 previous similar message [ 1305.576937] Lustre: Skipped 1 previous similar message [ 1310.501534] Lustre: server umount lustre-OST0000 complete [ 1315.809312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.810044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.813564] Lustre: Skipped 1 previous similar message [ 1318.027336] Lustre: server umount lustre-MDT0000 complete [ 1319.247930] LustreError: 82544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047643 with bad export cookie 5632757147128032217 [ 1319.250648] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.253048] LustreError: 82544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1319.354509] Lustre: server umount lustre-MDT0001 complete [ 1323.971652] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1325.049270] Key type lgssc unregistered [ 1325.175384] LNet: 86216:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.177949] LNet: Removed LNI 192.168.202.127@tcp [ 1325.471809] Key type .llcrypt unregistered [ 1325.472859] Key type ._llcrypt unregistered [ 1333.947317] alg: No test for adler32 (adler32-zlib) [ 1334.699326] Key type ._llcrypt registered [ 1334.700418] Key type .llcrypt registered [ 1334.743155] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1335.146570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.446890] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1335.533764] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1335.535746] LNet: Accept secure, port 988 [ 1337.119162] Key type lgssc registered [ 1337.502924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1340.218917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.364750] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.381226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.835156] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1345.623426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.727861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1347.019447] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1349.305049] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1351.016615] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1353.023584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1353.118358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.318784] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1356.196043] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 1356.477847] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1365.472467] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1365.474932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.479433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.481392] Lustre: Skipped 1 previous similar message [ 1369.492584] Lustre: server umount lustre-OST0000 complete [ 1371.615663] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1371.617828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.620958] Lustre: Skipped 1 previous similar message [ 1371.622245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.711976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.712495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.769034] Lustre: server umount lustre-MDT0000 complete [ 1377.927748] LustreError: 87655:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047701 with bad export cookie 12638005837660213081 [ 1377.930053] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.932127] LustreError: 87655:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1378.042012] Lustre: server umount lustre-MDT0001 complete [ 1382.423729] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1383.334540] Key type lgssc unregistered [ 1383.440291] LNet: 91327:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.442642] LNet: Removed LNI 192.168.202.127@tcp [ 1383.685719] Key type .llcrypt unregistered [ 1383.686662] Key type ._llcrypt unregistered [ 1392.053851] alg: No test for adler32 (adler32-zlib) [ 1392.805335] Key type ._llcrypt registered [ 1392.806345] Key type .llcrypt registered [ 1392.839277] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1393.233151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.357851] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1393.423322] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1393.424923] LNet: Accept secure, port 988 [ 1394.999095] Key type lgssc registered [ 1395.289433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1397.616161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.739287] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.753322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.035973] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1402.555247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.652302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.898377] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1406.040684] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1407.724675] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1409.727070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.820946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.102927] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1413.351102] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1413.923705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 1423.328794] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.330020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.335985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.337726] Lustre: Skipped 1 previous similar message [ 1428.239246] Lustre: server umount lustre-OST0000 complete [ 1433.568335] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1433.568443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.569036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.578012] Lustre: Skipped 2 previous similar messages [ 1435.518582] Lustre: server umount lustre-MDT0000 complete [ 1436.684903] LustreError: 92768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047760 with bad export cookie 14281112167037852252 [ 1436.687342] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.688835] LustreError: 92768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1436.790584] Lustre: server umount lustre-MDT0001 complete [ 1440.931813] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1441.869423] Key type lgssc unregistered [ 1441.978381] LNet: 96440:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.981495] LNet: Removed LNI 192.168.202.127@tcp [ 1442.255586] Key type .llcrypt unregistered [ 1442.256603] Key type ._llcrypt unregistered [ 1450.409394] alg: No test for adler32 (adler32-zlib) [ 1451.161353] Key type ._llcrypt registered [ 1451.162597] Key type .llcrypt registered [ 1451.198378] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1451.550434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.757632] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1451.837995] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1451.839426] LNet: Accept secure, port 988 [ 1453.415113] Key type lgssc registered [ 1453.771415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.321466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.823351] LustreError: 97872:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 1504.543184] LustreError: 97872:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 1535.263153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.594818] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1538.952081] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1540.771982] Lustre: server umount lustre-OST0000 complete [ 1543.481918] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:31:06 (1781047866) [ 1545.643941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.776848] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1545.787566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.149227] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1549.917649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.032283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1551.373049] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1553.710217] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1555.515640] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1557.666399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1557.776103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.136116] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1560.868985] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 1561.541957] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1570.274657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.275575] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1570.278865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1570.283514] Lustre: Skipped 1 previous similar message [ 1575.393357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1575.395111] Lustre: Skipped 1 previous similar message [ 1575.781257] Lustre: server umount lustre-OST0000 complete [ 1580.512770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.513340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.516376] Lustre: Skipped 2 previous similar messages [ 1585.631908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.634396] Lustre: Skipped 2 previous similar messages [ 1590.751919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.753966] Lustre: Skipped 1 previous similar message [ 1591.263133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1591.306905] Lustre: server umount lustre-MDT0000 complete [ 1592.529971] LustreError: 99449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047916 with bad export cookie 17470272363127227148 [ 1592.532479] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1592.535864] LustreError: 99449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1597.340612] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1598.303085] Key type lgssc unregistered [ 1598.416255] LNet: 103247:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.419165] LNet: Removed LNI 192.168.202.127@tcp [ 1598.714437] Key type .llcrypt unregistered [ 1598.715511] Key type ._llcrypt unregistered [ 1607.208740] alg: No test for adler32 (adler32-zlib) [ 1607.960352] Key type ._llcrypt registered [ 1607.961342] Key type .llcrypt registered [ 1608.002979] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1608.507563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.751661] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1608.831989] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1608.833844] LNet: Accept secure, port 988 [ 1610.415098] Key type lgssc registered [ 1610.745649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.388050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.536076] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.552440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.080287] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1618.909918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.038383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1620.618463] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1623.058309] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1624.771305] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1626.803687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.913536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.263183] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1629.987949] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 1630.489790] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1639.392573] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1639.393718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.395755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.400250] Lustre: Skipped 1 previous similar message [ 1644.176638] Lustre: server umount lustre-OST0000 complete [ 1649.632493] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1649.633110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.639917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1649.641760] Lustre: Skipped 2 previous similar messages [ 1651.459435] Lustre: server umount lustre-MDT0000 complete [ 1652.741197] LustreError: 104688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781047976 with bad export cookie 11267409385139911055 [ 1652.744273] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.744393] LustreError: 104688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1652.871035] Lustre: server umount lustre-MDT0001 complete [ 1657.087429] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1657.997764] Key type lgssc unregistered [ 1658.106284] LNet: 108364:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.108688] LNet: Removed LNI 192.168.202.127@tcp [ 1658.382948] Key type .llcrypt unregistered [ 1658.383811] Key type ._llcrypt unregistered [ 1664.769791] alg: No test for adler32 (adler32-zlib) [ 1665.520301] Key type ._llcrypt registered [ 1665.521399] Key type .llcrypt registered [ 1665.553650] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1665.927500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.170136] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1666.240337] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1666.242471] LNet: Accept secure, port 988 [ 1667.815111] Key type lgssc registered [ 1668.129458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1670.510805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.967364] LustreError: 109652:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 1718.687171] LustreError: 109652:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 1748.383186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.720138] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1752.144615] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1754.345983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.031196] LustreError: 15c-8: MGC192.168.202.127@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1771.035221] LustreError: 110636:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1771.037924] LustreError: 110636:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1771.040418] LustreError: 110636:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 1771.097209] Lustre: server umount lustre-MDT0000 complete [ 1771.098448] LustreError: 110636:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1780.191237] Lustre: 111069:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048097/real 1781048097] req@00000000928e3818 x1867564177164224/t0(0) o251->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781048103 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1780.245455] Lustre: server umount lustre-OST0000 complete [ 1783.231786] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 1786.012685] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1789.258643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1791.407270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1793.340059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1795.368180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1797.564657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1797.588557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.675191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1797.685703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1797.721037] Lustre: lustre-MDT0000: new disk, initializing [ 1797.742608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.748936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1799.128338] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1803.185728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1803.211659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.248105] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1803.266186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1803.270548] Lustre: Skipped 1 previous similar message [ 1803.314263] Lustre: lustre-MDT0001: new disk, initializing [ 1803.343080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1803.354054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1803.357223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1804.857270] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1808.602994] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.350145] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1812.796969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1812.820865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.823545] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1812.907733] Lustre: lustre-OST0000: new disk, initializing [ 1812.909423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1812.932961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.011746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1814.014804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1814.385736] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1818.089879] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1819.519462] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1819.588315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1820.943050] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1821.013468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1824.225810] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.226251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.226659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.228758] LustreError: Skipped 1 previous similar message [ 1824.235392] Lustre: Skipped 1 previous similar message [ 1827.797027] Lustre: server umount lustre-OST0000 complete [ 1829.344482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.344967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.348551] Lustre: Skipped 1 previous similar message [ 1829.350602] Lustre: Skipped 1 previous similar message [ 1834.463970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.465614] Lustre: Skipped 1 previous similar message [ 1835.138183] Lustre: server umount lustre-MDT0000 complete [ 1836.370063] LustreError: 114369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048160 with bad export cookie 14061266910688556071 [ 1836.372699] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1836.373713] LustreError: 114369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1839.116929] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:36:01 (1781048161) [ 1840.210990] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1845.354829] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 1848.068435] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1851.381590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.883553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.831405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1857.822750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.001434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.035399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.123285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1860.135148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1860.173939] Lustre: lustre-MDT0000: new disk, initializing [ 1860.198813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.205254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1861.583627] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1865.590246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1865.617517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.646544] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1865.661350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1865.663266] Lustre: Skipped 1 previous similar message [ 1865.697139] Lustre: lustre-MDT0001: new disk, initializing [ 1865.735697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1865.738546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1867.044957] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1870.732260] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.496733] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1874.598870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1874.620142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.623333] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1874.715449] Lustre: lustre-OST0000: new disk, initializing [ 1874.717541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1874.717589] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1874.748351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.750496] Lustre: Skipped 1 previous similar message [ 1876.273561] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1880.015614] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1880.038477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1880.041642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1881.437499] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1881.509987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1882.872185] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1882.950289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1885.152580] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.153819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.155683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.158375] Lustre: Skipped 1 previous similar message [ 1885.160960] Lustre: Skipped 1 previous similar message [ 1889.688387] Lustre: server umount lustre-OST0000 complete [ 1889.691213] Lustre: Skipped 1 previous similar message [ 1891.295778] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1891.299016] LustreError: Skipped 1 previous similar message [ 1891.300966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.391882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.392345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.393621] Lustre: Skipped 1 previous similar message [ 1898.239901] LustreError: 120636:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048222 with bad export cookie 14061266910688556911 [ 1898.242805] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.242933] LustreError: 120636:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1898.365039] Lustre: server umount lustre-MDT0001 complete [ 1898.366213] Lustre: Skipped 1 previous similar message [ 1901.042727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.178493] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.199783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.514759] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1905.200356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.647910] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1909.040854] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.865925] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1913.104159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1914.530884] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1916.915591] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1924.217562] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1925.601954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.602037] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1925.602533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1925.607884] Lustre: Skipped 1 previous similar message [ 1925.610542] LustreError: Skipped 1 previous similar message [ 1927.254379] Lustre: server umount lustre-OST0000 complete [ 1929.799591] LustreError: 125026:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048253 with bad export cookie 14061266910688557751 [ 1929.799746] LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [ 1929.803433] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.582728] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1941.600455] Key type lgssc unregistered [ 1941.714267] LNet: 128750:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.716560] LNet: Removed LNI 192.168.202.127@tcp [ 1942.000823] Key type .llcrypt unregistered [ 1942.001986] Key type ._llcrypt unregistered [ 1948.158628] alg: No test for adler32 (adler32-zlib) [ 1948.910351] Key type ._llcrypt registered [ 1948.911696] Key type .llcrypt registered [ 1948.952893] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 1954.863726] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1955.173697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.410328] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 1955.482436] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1955.484226] LNet: Accept secure, port 988 [ 1957.063110] Key type lgssc registered [ 1957.435416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.370952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.518033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1964.407867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.329710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.560592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.581811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.662738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1969.674137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1969.710725] Lustre: lustre-MDT0000: new disk, initializing [ 1969.733610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.739505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1971.114147] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1975.150639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1975.172210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.200857] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1975.223871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1975.226028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1975.226759] Lustre: Skipped 1 previous similar message [ 1975.263803] Lustre: lustre-MDT0001: new disk, initializing [ 1975.294257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.308635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1975.312702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1976.647256] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1980.363638] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.266637] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1984.437314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1984.465036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.469089] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1984.563359] Lustre: lustre-OST0000: new disk, initializing [ 1984.565141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1984.595254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.944303] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 1989.668156] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1990.117506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1990.120059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1991.010532] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1991.073402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1992.325183] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1992.391745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1995.234289] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.234903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.236166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.237903] LustreError: Skipped 1 previous similar message [ 1995.242648] Lustre: Skipped 1 previous similar message [ 1999.192035] Lustre: server umount lustre-OST0000 complete [ 2000.863905] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.866384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.870515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.872950] Lustre: Skipped 1 previous similar message [ 2005.471985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.472409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.530797] Lustre: server umount lustre-MDT0000 complete [ 2007.714049] LustreError: 132015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048331 with bad export cookie 1572217491449049430 [ 2007.716160] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.719295] LustreError: 132015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2007.833876] Lustre: server umount lustre-MDT0001 complete [ 2010.539336] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:38:53 (1781048333) [ 2012.694408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.845853] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.864388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.462789] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2017.265364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.386966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2018.769627] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2021.267735] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2023.107051] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.488738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.488807] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2027.489570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.489574] Lustre: Skipped 1 previous similar message [ 2027.493709] Lustre: Skipped 1 previous similar message [ 2037.727869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.729544] Lustre: Skipped 4 previous similar messages [ 2038.239105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2038.280072] Lustre: server umount lustre-MDT0000 complete [ 2039.506939] LustreError: 138246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048363 with bad export cookie 1572217491449050270 [ 2039.509866] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.510747] LustreError: 138246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2042.187972] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:39:24 (1781048364) [ 2044.328993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2061.791718] LustreError: 139092:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 2072.031493] LustreError: 139092:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2101.727287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.029138] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2105.369751] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2106.057347] Lustre: server umount lustre-OST0000 complete [ 2106.058894] Lustre: Skipped 1 previous similar message [ 2108.527619] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:40:31 (1781048431) [ 2110.595686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.730481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.164362] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2114.832449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.305055] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2118.645722] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.509679] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2122.680818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.788916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.791617] Lustre: Skipped 2 previous similar messages [ 2124.145971] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2126.703578] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2135.009405] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.010238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.014509] Lustre: Skipped 1 previous similar message [ 2135.018677] Lustre: Skipped 2 previous similar messages [ 2140.747414] Lustre: server umount lustre-MDT0000 complete [ 2142.013524] LustreError: 140474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048465 with bad export cookie 1572217491449050921 [ 2142.016988] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.017733] LustreError: 140474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2149.855198] Lustre: 143543:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048467/real 1781048467] req@00000000968c3b1f x1867564480234496/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781048473 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2152.473928] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:41:15 (1781048475) [ 2154.515805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.659951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.006340] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2158.626344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.736201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.738482] Lustre: Skipped 1 previous similar message [ 2160.057192] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2162.355282] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.170600] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2166.369725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2167.971036] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2169.574076] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 2170.378726] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2171.849883] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2171.936399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2173.356028] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2174.451614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2179.040886] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.043176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.046337] Lustre: Skipped 1 previous similar message [ 2179.048530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.050273] Lustre: Skipped 4 previous similar messages [ 2189.279096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2189.333807] Lustre: server umount lustre-OST0000 complete [ 2189.335178] Lustre: Skipped 2 previous similar messages [ 2197.862958] LustreError: 144997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048521 with bad export cookie 1572217491449051964 [ 2197.864652] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.868073] LustreError: 144997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2200.477232] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:42:03 (1781048523) [ 2202.859981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2219.359534] LustreError: 148544:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 2230.623360] LustreError: 148544:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2259.295221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.297050] Lustre: Skipped 1 previous similar message [ 2260.596933] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2262.971767] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2265.089526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.811533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.860337] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 2280.204821] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2282.988865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.458498] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2286.840435] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2288.655915] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2290.117718] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2290.193923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2291.681607] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2291.772539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2293.216533] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2293.218767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.221778] Lustre: Skipped 3 previous similar messages [ 2293.223262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2293.225027] Lustre: Skipped 6 previous similar messages [ 2298.634248] Lustre: server umount lustre-MDT0000 complete [ 2298.636426] Lustre: Skipped 2 previous similar messages [ 2299.868540] LustreError: 149570:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048623 with bad export cookie 1572217491449052993 [ 2299.871581] LustreError: 149570:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2299.871968] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.551155] Lustre: 152257:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048625/real 1781048625] req@000000005952e8db x1867564480256000/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781048631 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2310.186696] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:43:52 (1781048632) [ 2312.493799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.015430] LustreError: 152900:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 2341.279349] LustreError: 152900:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2372.455295] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2374.935408] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2377.162164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.413720] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.438497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.440080] Lustre: Skipped 3 previous similar messages [ 2387.462054] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 2388.898550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2391.800605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.421564] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2395.833286] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2397.707791] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2399.849654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.880038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.882500] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2399.937164] Lustre: lustre-OST0001: new disk, initializing [ 2399.938921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2401.354441] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2403.186355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2403.189393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2405.059501] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2406.532726] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2406.604510] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2408.000655] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2408.087646] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2412.512420] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.514624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.517631] Lustre: Skipped 1 previous similar message [ 2423.263098] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2423.281199] Lustre: server umount lustre-OST0000 complete [ 2423.282406] Lustre: Skipped 2 previous similar messages [ 2423.779796] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.784538] LustreError: Skipped 3 previous similar messages [ 2428.899562] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.902310] LustreError: Skipped 2 previous similar messages [ 2428.902614] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2428.905421] Lustre: Skipped 10 previous similar messages [ 2446.303198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2447.605156] LustreError: 152908:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048771 with bad export cookie 1572217491449053910 [ 2447.607724] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.608161] LustreError: 152908:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2449.594637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.840673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.060723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2453.268764] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2455.235476] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:46:17 (1781048777) [ 2455.786182] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2456.273314] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2456.801603] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:46:19 (1781048779) [ 2458.856457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.948845] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2458.958039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2460.290962] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2462.871911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.903454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2462.913135] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2464.321068] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2466.739574] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.441983] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2470.475454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.537472] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2471.889924] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2473.603750] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 2474.174141] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2475.543884] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2475.609328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2477.081321] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2477.167573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2479.074539] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2479.075376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.078743] LustreError: Skipped 2 previous similar messages [ 2479.081521] Lustre: Skipped 5 previous similar messages [ 2493.722290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.226602] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2495.906989] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.910603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.910677] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2495.912225] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 2495.919175] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 2495.919776] Lustre: Skipped 1 previous similar message [ 2497.640825] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2499.304443] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2499.396943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2500.822459] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2500.901590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2502.778749] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff934b87e9b800.ost_server_uuid 40 [ 2503.261187] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff934b87e9b800.ost_server_uuid in FULL state after 0 sec [ 2503.315462] LustreError: 159480:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 192.168.202.27@tcp) returned error from blocking AST (req@0000000045c39039 x1867564480294528 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003040e28f/0x15d1a37a7f7a0be3 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.27@tcp remote: 0x1474d85370ddeeb5 expref: 6 pid: 160597 timeout: 2603 lvb_type: 0 [ 2503.325362] LustreError: 138-a: lustre-MDT0000: A client on nid 192.168.202.27@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2503.328454] LustreError: 159470:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.27@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003040e28f/0x15d1a37a7f7a0be3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.27@tcp remote: 0x1474d85370ddeeb5 expref: 7 pid: 160597 timeout: 0 lvb_type: 0 [ 2526.175173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2527.486445] LustreError: 159466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048851 with bad export cookie 1572217491449055058 [ 2527.489738] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.490265] LustreError: 159466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2538.541469] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2539.568242] Key type lgssc unregistered [ 2539.695415] LNet: 165865:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.698965] LNet: Removed LNI 192.168.202.127@tcp [ 2540.004945] Key type .llcrypt unregistered [ 2540.006205] Key type ._llcrypt unregistered [ 2548.885498] alg: No test for adler32 (adler32-zlib) [ 2549.636395] Key type ._llcrypt registered [ 2549.637349] Key type .llcrypt registered [ 2549.676079] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2550.064915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.348731] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2550.430675] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2550.432938] LNet: Accept secure, port 988 [ 2552.015128] Key type lgssc registered [ 2552.403733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2555.212427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.381722] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.398809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.838440] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2560.771658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.895758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2562.321766] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2565.051228] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2566.887200] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2568.984427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.090540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.446068] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2572.198081] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2572.958688] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2579.807271] Lustre: Failing over lustre-MDT0000 [ 2579.890276] Lustre: server umount lustre-MDT0000 complete [ 2580.449066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.450115] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.452259] Lustre: Skipped 2 previous similar messages [ 2580.455375] LustreError: Skipped 1 previous similar message [ 2585.568393] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.577239] LustreError: Skipped 3 previous similar messages [ 2587.615148] Lustre: 166638:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048904/real 1781048904] req@00000000f53adc7a x1867565104112896/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781048911 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2587.634718] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.641722] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.647646] LustreError: Skipped 2 previous similar messages [ 2590.854154] LustreError: 167304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.27@tcp arrived at 1781048914 with bad export cookie 13147247027385710144 [ 2590.860418] LustreError: 167304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2592.741290] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.749900] LustreError: Skipped 2 previous similar messages [ 2594.778067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.005594] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.013551] LustreError: Skipped 6 previous similar messages [ 2605.098205] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0xb6746783a9a2b2d5 to 0xb6746783a9a2b6b0 [ 2605.130105] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 2605.636030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.708141] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2610.669604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2610.680899] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2611.754285] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2614.428491] Lustre: lustre-MDT0000: Denying connection for new client 86bccbbd-48fc-4f9c-add3-20ee01c87316 (at 192.168.202.27@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 2626.034098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.053748] Lustre: Skipped 1 previous similar message [ 2626.054837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.065676] Lustre: Skipped 1 previous similar message [ 2629.496507] Lustre: server umount lustre-OST0000 complete [ 2633.310792] LustreError: 171176:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2633.315754] Lustre: 170472:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2633.318549] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 2633.342546] Lustre: 170472:0:(mdt_handler.c:7532:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 2636.265447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.283629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.391190] Lustre: 166635:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781048934/real 1781048934] req@0000000006b23112 x1867565104118720/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 2 to 1 dl 1781048963 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'' [ 2639.406968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.538455] Lustre: server umount lustre-MDT0000 complete [ 2641.446732] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.465287] LustreError: Skipped 6 previous similar messages [ 2641.959894] LustreError: 169971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781048965 with bad export cookie 13147247027385710256 [ 2641.967642] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.974143] LustreError: 169971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2648.378452] Lustre: server umount lustre-MDT0001 complete [ 2656.574856] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2658.242829] Key type lgssc unregistered [ 2658.384423] LNet: 171981:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.389611] LNet: Removed LNI 192.168.202.127@tcp [ 2658.902747] Key type .llcrypt unregistered [ 2658.903889] Key type ._llcrypt unregistered [ 2670.248048] alg: No test for adler32 (adler32-zlib) [ 2670.999349] Key type ._llcrypt registered [ 2671.000715] Key type .llcrypt registered [ 2671.057191] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2671.615151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.760815] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2671.870052] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2671.872702] LNet: Accept secure, port 988 [ 2673.463094] Key type lgssc registered [ 2673.892877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.314425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.514106] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.547836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.343366] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2683.875085] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.677430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.871142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.793062] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2690.879990] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2694.039155] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2697.402061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2697.560891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.922806] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2703.955148] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2708.843340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 2714.079976] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.083574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.088935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2714.090889] Lustre: Skipped 1 previous similar message [ 2714.097846] Lustre: Skipped 1 previous similar message [ 2717.360854] Lustre: server umount lustre-OST0000 complete [ 2719.202075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.208642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2719.210880] Lustre: Skipped 1 previous similar message [ 2724.319985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2724.322811] Lustre: Skipped 1 previous similar message [ 2725.130613] Lustre: server umount lustre-MDT0000 complete [ 2726.956520] LustreError: 176290:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049050 with bad export cookie 17456556949422424434 [ 2726.964354] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.119700] Lustre: server umount lustre-MDT0001 complete [ 2733.044912] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2734.368798] Key type lgssc unregistered [ 2734.528709] LNet: 177093:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.532523] LNet: Removed LNI 192.168.202.127@tcp [ 2734.938716] Key type .llcrypt unregistered [ 2734.940608] Key type ._llcrypt unregistered [ 2751.567115] alg: No test for adler32 (adler32-zlib) [ 2752.319384] Key type ._llcrypt registered [ 2752.320509] Key type .llcrypt registered [ 2752.379541] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2752.804859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2753.101744] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2753.225157] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2753.227745] LNet: Accept secure, port 988 [ 2754.824229] Key type lgssc registered [ 2755.321126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.963753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.179095] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.202945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.031748] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2765.284810] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.209212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.375559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.234415] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2771.494448] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2773.862170] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2776.485915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.647446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.339166] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2781.851355] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2791.013455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 2796.512911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.514424] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2796.519229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2796.524039] Lustre: Skipped 1 previous similar message [ 2800.284953] Lustre: server umount lustre-OST0000 complete [ 2806.756192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.757092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2806.765200] Lustre: Skipped 2 previous similar messages [ 2806.773253] Lustre: Skipped 1 previous similar message [ 2808.397584] Lustre: server umount lustre-MDT0000 complete [ 2810.142950] LustreError: 178835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049133 with bad export cookie 4518883974785721056 [ 2810.145892] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.149251] LustreError: 178835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2810.309735] Lustre: server umount lustre-MDT0001 complete [ 2816.252849] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2817.522086] Key type lgssc unregistered [ 2817.675393] LNet: 182525:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.681867] LNet: Removed LNI 192.168.202.127@tcp [ 2818.090434] Key type .llcrypt unregistered [ 2818.091864] Key type ._llcrypt unregistered [ 2828.865581] alg: No test for adler32 (adler32-zlib) [ 2829.619350] Key type ._llcrypt registered [ 2829.620627] Key type .llcrypt registered [ 2829.677517] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2830.041751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.318407] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2830.433487] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2830.435849] LNet: Accept secure, port 988 [ 2832.039170] Key type lgssc registered [ 2832.586168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.948688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.149763] Lustre: *** cfs_fail_loc=135, val=0*** [ 2839.202154] LustreError: 184096:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 2839.205430] LustreError: 184096:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.127@tcp: cfg command failed: rc = -2 [ 2839.212740] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2839.212740] [ 2839.218615] LustreError: 15c-8: MGC192.168.202.127@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2839.226197] LustreError: 184058:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2839.235858] LustreError: 184058:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2839.244291] LustreError: 184058:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2845.663545] Lustre: 184058:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781049163/real 1781049163] req@000000009c8ad11e x1867565397705536/t0(0) o251->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781049169 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2845.747196] Lustre: server umount lustre-MDT0000 complete [ 2845.749317] LustreError: 184058:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2848.579064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.778090] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.797793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.496130] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2853.857219] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.049579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.194443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.948762] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2859.390697] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2861.930334] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2864.612747] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2864.614273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.624772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.632436] Lustre: Skipped 1 previous similar message [ 2868.942748] Lustre: server umount lustre-MDT0000 complete [ 2869.729525] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.734161] LustreError: Skipped 1 previous similar message [ 2870.453045] LustreError: 186147:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049194 with bad export cookie 7101148228392614457 [ 2870.455355] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.457301] LustreError: 186147:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2870.568061] Lustre: server umount lustre-MDT0001 complete [ 2874.742802] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2875.846422] Key type lgssc unregistered [ 2875.961433] LNet: 186949:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.969437] LNet: Removed LNI 192.168.202.127@tcp [ 2876.267813] Key type .llcrypt unregistered [ 2876.269466] Key type ._llcrypt unregistered [ 2884.889263] alg: No test for adler32 (adler32-zlib) [ 2885.641454] Key type ._llcrypt registered [ 2885.642799] Key type .llcrypt registered [ 2885.688269] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2885.814869] Key type .llcrypt unregistered [ 2885.816189] Key type ._llcrypt unregistered [ 2893.143358] alg: No test for adler32 (adler32-zlib) [ 2893.895442] Key type ._llcrypt registered [ 2893.897307] Key type .llcrypt registered [ 2893.965398] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2894.380939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2894.651656] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 2894.755337] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2894.756936] LNet: Accept secure, port 988 [ 2896.351391] Key type lgssc registered [ 2896.923516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2900.150651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.663308] LustreError: 189200:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 1: rc = -5 [ 2948.383314] LustreError: 189200:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2978.079203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.549272] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 2982.108121] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2984.477327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.733792] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.762762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.789488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 3002.156072] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3005.071133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.194857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3006.575187] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3009.115780] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.949837] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3011.897200] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3020.768284] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3020.771046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.775249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.279631] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.282096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.286237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.808233] Lustre: server umount lustre-OST0000 complete [ 3025.887833] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.890463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.895145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.896972] Lustre: Skipped 1 previous similar message [ 3030.217028] Lustre: server umount lustre-MDT0000 complete [ 3031.492269] LustreError: 189208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049355 with bad export cookie 9406345183798803238 [ 3031.495365] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.495970] LustreError: 189208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3031.633605] Lustre: server umount lustre-MDT0001 complete [ 3035.499660] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3036.611480] Key type lgssc unregistered [ 3036.734349] LNet: 193236:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.737340] LNet: Removed LNI 192.168.202.127@tcp [ 3037.053409] Key type .llcrypt unregistered [ 3037.054635] Key type ._llcrypt unregistered [ 3045.800449] alg: No test for adler32 (adler32-zlib) [ 3046.552468] Key type ._llcrypt registered [ 3046.553512] Key type .llcrypt registered [ 3046.593363] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3047.017626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3047.288300] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3047.375118] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3047.377601] LNet: Accept secure, port 988 [ 3048.959130] Key type lgssc registered [ 3049.317410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.859720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.993155] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.007605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.361481] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3057.127050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.242741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.693491] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3061.124187] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3062.888637] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3064.984258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.102835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.443631] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3068.197711] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 3068.729257] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3070.823537] Lustre: Failing over lustre-MDT0000 [ 3070.889943] Lustre: server umount lustre-MDT0000 complete [ 3072.288826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.290304] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.294409] Lustre: Skipped 2 previous similar messages [ 3072.299392] LustreError: Skipped 1 previous similar message [ 3075.324477] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.328287] LustreError: Skipped 1 previous similar message [ 3077.600608] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.604473] LustreError: Skipped 2 previous similar messages [ 3079.391214] Lustre: 194006:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781049396/real 1781049396] req@000000004fd0b34c x1867565625253952/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781049403 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3079.398851] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.495541] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.499834] LustreError: Skipped 4 previous similar messages [ 3084.091898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.794053] Lustre: Evicted from MGS (at 192.168.202.127@tcp) after server handle changed from 0x96bdd33c10a1ae2 to 0x96bdd33c10a1ea8 [ 3085.800432] Lustre: MGC192.168.202.127@tcp: Connection restored to 192.168.202.127@tcp (at 0@lo) [ 3085.882916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.918659] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3087.216067] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3087.605242] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3088.692835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 3090.918379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3090.923443] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3090.939603] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 3097.072139] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 3108.319757] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.322032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.327273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.881357] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3110.883830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.887663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3112.404293] Lustre: server umount lustre-OST0000 complete [ 3116.512891] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.513145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.513602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.513607] Lustre: Skipped 2 previous similar messages [ 3116.522547] Lustre: Skipped 1 previous similar message [ 3119.749286] Lustre: server umount lustre-MDT0000 complete [ 3120.951309] LustreError: 194676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049444 with bad export cookie 678879383201128104 [ 3120.953028] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.956274] LustreError: 194676:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3125.640630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3126.668639] Key type lgssc unregistered [ 3126.792531] LNet: 199937:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3126.795559] LNet: Removed LNI 192.168.202.127@tcp [ 3127.100618] Key type .llcrypt unregistered [ 3127.102165] Key type ._llcrypt unregistered [ 3135.945685] alg: No test for adler32 (adler32-zlib) [ 3136.697318] Key type ._llcrypt registered [ 3136.698619] Key type .llcrypt registered [ 3136.732198] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3137.063830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3137.270524] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3137.343883] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3137.345486] LNet: Accept secure, port 988 [ 3138.919088] Key type lgssc registered [ 3139.223806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3141.798113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.928539] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.943363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.357343] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3147.199623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.307786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3148.664875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3151.161188] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3153.027586] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.302224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.423673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.786414] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3158.502661] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 3159.185383] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3160.708257] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3169.296388] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3177.275739] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3188.193253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.194428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.197097] Lustre: Skipped 1 previous similar message [ 3188.197516] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.199063] Lustre: Skipped 1 previous similar message [ 3192.534070] Lustre: server umount lustre-OST0000 complete [ 3198.432519] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3198.432581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.433106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3198.440328] Lustre: Skipped 1 previous similar message [ 3203.551857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.553684] Lustre: Skipped 2 previous similar messages [ 3208.159122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3208.203615] Lustre: server umount lustre-MDT0000 complete [ 3209.455140] LustreError: 203352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049533 with bad export cookie 9054654983257490967 [ 3209.458620] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.460324] LustreError: 203352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3209.601360] Lustre: server umount lustre-MDT0001 complete [ 3214.195906] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3215.212612] Key type lgssc unregistered [ 3215.325333] LNet: 205225:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3215.327967] LNet: Removed LNI 192.168.202.127@tcp [ 3215.606406] Key type .llcrypt unregistered [ 3215.608129] Key type ._llcrypt unregistered [ 3224.499541] alg: No test for adler32 (adler32-zlib) [ 3225.251308] Key type ._llcrypt registered [ 3225.252610] Key type .llcrypt registered [ 3225.297307] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3225.720454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3225.940536] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3226.010046] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3226.011665] LNet: Accept secure, port 988 [ 3227.583080] Key type lgssc registered [ 3227.884705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3230.408786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.544869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.559835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.865793] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3235.537247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.651496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3237.029543] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3239.391356] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.204629] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3243.323421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.434790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.790763] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3246.503101] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 3247.149089] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3249.484301] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3255.739312] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3266.147086] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3273.714075] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3281.377494] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3281.377772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.384063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3281.385881] Lustre: Skipped 1 previous similar message [ 3286.100539] Lustre: server umount lustre-OST0000 complete [ 3291.616199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.616838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.619288] Lustre: Skipped 2 previous similar messages [ 3293.443172] Lustre: server umount lustre-MDT0000 complete [ 3294.604869] LustreError: 206666:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049618 with bad export cookie 7336178633956509374 [ 3294.607299] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.607591] LustreError: 206666:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3294.722034] Lustre: server umount lustre-MDT0001 complete [ 3299.113034] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3300.023640] Key type lgssc unregistered [ 3300.148279] LNet: 212208:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3300.150909] LNet: Removed LNI 192.168.202.127@tcp [ 3300.431429] Key type .llcrypt unregistered [ 3300.434052] Key type ._llcrypt unregistered [ 3308.901260] alg: No test for adler32 (adler32-zlib) [ 3309.652307] Key type ._llcrypt registered [ 3309.653358] Key type .llcrypt registered [ 3309.690803] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3309.978847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3310.202847] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3310.281921] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3310.283380] LNet: Accept secure, port 988 [ 3311.863097] Key type lgssc registered [ 3312.334411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3315.642501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.775289] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.789030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.195424] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3321.417638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.598268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.149888] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3326.016361] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3328.384226] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3330.625985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.744667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.210879] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3332.773957] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 3334.586736] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3337.870931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.893370] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3337.953714] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.284261] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3341.671617] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3345.283426] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3355.870424] Lustre: Permanently deactivating lustre-OST0001 [ 3355.875820] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3361.789903] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3373.786723] Lustre: Permanently reactivating lustre-OST0001 [ 3373.793731] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3373.796446] Lustre: Skipped 2 previous similar messages [ 3382.766791] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.771619] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3387.051153] Lustre: server umount lustre-OST0001 complete [ 3387.873717] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.880073] LustreError: Skipped 1 previous similar message [ 3388.384260] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.388795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.396039] Lustre: Skipped 1 previous similar message [ 3388.399647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.403179] Lustre: Skipped 1 previous similar message [ 3389.407957] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.409984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.413317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.992694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.994608] Lustre: Skipped 1 previous similar message [ 3392.994671] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.999442] LustreError: Skipped 1 previous similar message [ 3394.455453] Lustre: server umount lustre-OST0000 complete [ 3398.112627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.120290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.121921] Lustre: Skipped 1 previous similar message [ 3401.800239] Lustre: server umount lustre-MDT0000 complete [ 3402.995034] LustreError: 216581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049726 with bad export cookie 7518073911324315926 [ 3402.997926] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.000580] LustreError: 216581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3407.540289] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3408.503716] Key type lgssc unregistered [ 3408.605431] LNet: 218808:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.608569] LNet: Removed LNI 192.168.202.127@tcp [ 3408.905980] Key type .llcrypt unregistered [ 3408.907661] Key type ._llcrypt unregistered [ 3419.513828] alg: No test for adler32 (adler32-zlib) [ 3420.265351] Key type ._llcrypt registered [ 3420.266338] Key type .llcrypt registered [ 3420.308871] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 3426.011177] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3426.458779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3426.586313] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3426.665479] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3426.667142] LNet: Accept secure, port 988 [ 3428.247091] Key type lgssc registered [ 3428.577831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3431.405922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3433.451640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.334047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3437.282644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3439.442290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3439.466053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.563041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3440.573865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3440.610210] Lustre: lustre-MDT0000: new disk, initializing [ 3440.632996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.638712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3442.007940] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3446.105583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3446.129047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.161023] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3446.177495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3446.179186] Lustre: Skipped 1 previous similar message [ 3446.212866] Lustre: lustre-MDT0001: new disk, initializing [ 3446.242835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.256838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3446.262113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3447.596188] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3451.355919] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3453.212283] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3455.461909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.488049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.492675] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3455.587697] Lustre: lustre-OST0000: new disk, initializing [ 3455.589390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3455.614663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.082607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3456.085481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3457.005515] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3460.774254] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3462.176468] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3462.249321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3463.549119] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3463.623552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3466.322442] Lustre: server umount lustre-OST0000 complete [ 3471.840375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.840711] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3471.841172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.843979] Lustre: Skipped 1 previous similar message [ 3473.887199] Lustre: 226575:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781049791/real 1781049791] req@0000000048aab036 x1867566022669696/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781049797 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3473.887249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.893396] Lustre: 226575:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3473.900534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.902204] Lustre: Skipped 2 previous similar messages [ 3473.954527] Lustre: server umount lustre-MDT0000 complete [ 3475.139092] LustreError: 223746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781049798 with bad export cookie 2260446876523180347 [ 3475.142136] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.144095] LustreError: 223746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3475.256446] Lustre: server umount lustre-MDT0001 complete [ 3477.812730] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 20:03:20 (1781049800) [ 3480.029771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.167268] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.186508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.508239] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3484.231213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.341873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.876647] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3488.305197] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.278166] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.508095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.623928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.969414] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3496.344266] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.856393] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3503.332853] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3509.842579] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3525.004261] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3525.008204] Lustre: Skipped 1 previous similar message [ 3545.573643] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3545.576056] Lustre: Skipped 2 previous similar messages [ 3580.026190] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3580.028879] Lustre: Skipped 3 previous similar messages [ 3648.252233] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3648.255216] Lustre: Skipped 8 previous similar messages [ 3658.721701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.722780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.725130] Lustre: Skipped 1 previous similar message [ 3658.728109] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.612094] Lustre: server umount lustre-OST0000 complete [ 3663.840130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.840230] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3663.840763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.840767] Lustre: Skipped 1 previous similar message [ 3663.843741] Lustre: Skipped 1 previous similar message [ 3668.960474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.962111] Lustre: Skipped 2 previous similar messages [ 3678.175108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3678.214260] Lustre: server umount lustre-MDT0000 complete [ 3679.463944] LustreError: 230043:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050003 with bad export cookie 2260446876523181187 [ 3679.466705] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.466848] LustreError: 230043:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3684.162546] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3685.402727] Key type lgssc unregistered [ 3685.578469] LNet: 232348:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.583156] LNet: Removed LNI 192.168.202.127@tcp [ 3686.072120] Key type .llcrypt unregistered [ 3686.074369] Key type ._llcrypt unregistered [ 3695.319930] alg: No test for adler32 (adler32-zlib) [ 3696.071326] Key type ._llcrypt registered [ 3696.072269] Key type .llcrypt registered [ 3696.108566] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3696.461866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3696.652960] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3696.723525] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3696.725115] LNet: Accept secure, port 988 [ 3698.303148] Key type lgssc registered [ 3698.635371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3701.232056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.369532] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3702.382934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.706344] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3707.062620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.195788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3708.816257] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3711.507201] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3713.362767] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3715.619185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.738464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.249744] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all 8 [ 3719.711843] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3734.496568] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3734.498123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3734.499055] LustreError: Skipped 1 previous similar message [ 3734.502647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3734.505463] Lustre: Skipped 1 previous similar message [ 3735.508668] Lustre: server umount lustre-OST0000 complete [ 3738.079686] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3738.081726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.084677] Lustre: Skipped 1 previous similar message [ 3738.086121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.616296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3739.616676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.684222] Lustre: server umount lustre-MDT0000 complete [ 3744.736055] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.739861] LustreError: Skipped 1 previous similar message [ 3744.829841] LustreError: 236511:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781050068 with bad export cookie 2095911899439514283 [ 3744.832588] LustreError: 166-1: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.833358] LustreError: 236511:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3744.944793] Lustre: server umount lustre-MDT0001 complete [ 3749.641488] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3750.589519] Key type lgssc unregistered [ 3750.705472] LNet: 237618:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3750.708154] LNet: Removed LNI 192.168.202.127@tcp [ 3750.998849] Key type .llcrypt unregistered [ 3751.000127] Key type ._llcrypt unregistered [ 3758.762217] alg: No test for adler32 (adler32-zlib) [ 3759.513278] Key type ._llcrypt registered [ 3759.514266] Key type .llcrypt registered [ 3759.558550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3759.648558] Key type .llcrypt unregistered [ 3759.649980] Key type ._llcrypt unregistered [ 3767.564762] alg: No test for adler32 (adler32-zlib) [ 3768.316377] Key type ._llcrypt registered [ 3768.317343] Key type .llcrypt registered [ 3768.354460] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 3774.368529] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3774.672324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.905390] Lustre: Lustre: Build Version: 2.15.8_5_gc131e65 [ 3774.974685] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3774.976319] LNet: Accept secure, port 988 [ 3776.551064] Key type lgssc registered [ 3776.877684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.342051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3783.062600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3785.247177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3787.456747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3787.990159] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3651 sec ========== 20:08:30 (1781050110) [ 3801.772335] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3804.306749] Key type lgssc unregistered [ 3804.557640] LNet: 243838:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3804.561346] LNet: Removed LNI 192.168.202.127@tcp [ 3805.052944] Key type .llcrypt unregistered [ 3805.054774] Key type ._llcrypt unregistered