[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501985606 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003092] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.007901] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009000] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.011971] *** VALIDATE selinux *** [ 0.018657] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023213] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024235] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025302] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026169] *** VALIDATE tmpfs *** [ 0.028502] *** VALIDATE proc *** [ 0.029271] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033075] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035032] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039135] debug: unmapping init [mem 0xffffffff8be59000-0xffffffff8be60fff] [ 0.041156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046009] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.050361] rcu: Hierarchical SRCU implementation. [ 0.052506] smp: Bringing up secondary CPUs ... [ 0.053545] x86: Booting SMP configuration: [ 0.054015] .... node #0, CPUs: #1 #2 #3 [ 0.057137] smp: Brought up 1 node, 4 CPUs [ 0.059008] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.283217] node 0 deferred pages initialised in 221ms [ 0.288011] devtmpfs: initialized [ 0.289309] x86/mm: Memory block size: 128MB [ 0.291733] gcov: version magic: 0x41383552 [ 0.293472] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.294091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.295000] pinctrl core: initialized pinctrl subsystem [ 0.298232] [ 0.298872] ************************************************************* [ 0.300012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302008] ** ** [ 0.303006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.305008] ** ** [ 0.307017] ** This means that this kernel is built to expose internal ** [ 0.309007] ** IOMMU data structures, which may compromise security on ** [ 0.311009] ** your system. ** [ 0.312010] ** ** [ 0.315011] ** If you see this message and you are not debugging the ** [ 0.317007] ** kernel, report this immediately to your vendor! ** [ 0.318006] ** ** [ 0.319006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.320005] ************************************************************* [ 0.321678] NET: Registered protocol family 16 [ 0.323404] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.325048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.328043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.330275] cpuidle: using governor menu [ 0.332000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.334415] PCI: Using configuration type 1 for base access [ 0.336103] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.343119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.345034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.349090] cryptd: max_cpu_qlen set to 1000 [ 0.351248] ACPI: Added _OSI(Module Device) [ 0.353014] ACPI: Added _OSI(Processor Device) [ 0.355012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.356010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.360614] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.366471] ACPI: Interpreter enabled [ 0.368051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.369007] ACPI: Using IOAPIC for interrupt routing [ 0.370102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372264] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.380560] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.382022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.384013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.386075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.391293] acpiphp: Slot [2] registered [ 0.392075] acpiphp: Slot [3] registered [ 0.393062] acpiphp: Slot [4] registered [ 0.394075] acpiphp: Slot [5] registered [ 0.396083] acpiphp: Slot [6] registered [ 0.397101] acpiphp: Slot [7] registered [ 0.398205] acpiphp: Slot [8] registered [ 0.400087] acpiphp: Slot [9] registered [ 0.402136] acpiphp: Slot [10] registered [ 0.403110] acpiphp: Slot [11] registered [ 0.404062] acpiphp: Slot [12] registered [ 0.406260] acpiphp: Slot [13] registered [ 0.407065] acpiphp: Slot [14] registered [ 0.408097] acpiphp: Slot [15] registered [ 0.410089] acpiphp: Slot [16] registered [ 0.411105] acpiphp: Slot [17] registered [ 0.413297] acpiphp: Slot [18] registered [ 0.414084] acpiphp: Slot [19] registered [ 0.415076] acpiphp: Slot [20] registered [ 0.416067] acpiphp: Slot [21] registered [ 0.418088] acpiphp: Slot [22] registered [ 0.419262] acpiphp: Slot [23] registered [ 0.421094] acpiphp: Slot [24] registered [ 0.422081] acpiphp: Slot [25] registered [ 0.423100] acpiphp: Slot [26] registered [ 0.424086] acpiphp: Slot [27] registered [ 0.426109] acpiphp: Slot [28] registered [ 0.427108] acpiphp: Slot [29] registered [ 0.428000] acpiphp: Slot [30] registered [ 0.429142] acpiphp: Slot [31] registered [ 0.431118] PCI host bridge to bus 0000:00 [ 0.432024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.437024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.439054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.441044] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.444021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.446227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.449000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.454995] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.464820] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.469865] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.471010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.473012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.475029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.477510] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.480740] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.483044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.487367] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.492014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.503013] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.507022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.513108] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.549089] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.562088] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.605025] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.622853] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.668028] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.681073] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.687016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.694015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.712029] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.723530] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.733017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.747026] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.758443] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.765024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.771015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.805026] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.818000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.819315] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.822272] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.824322] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.826250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.831113] iommu: Default domain type: Passthrough [ 0.832000] SCSI subsystem initialized [ 0.832000] ACPI: bus type USB registered [ 0.833116] usbcore: registered new interface driver usbfs [ 0.834090] usbcore: registered new interface driver hub [ 0.835072] usbcore: registered new device driver usb [ 0.837158] pps_core: LinuxPPS API ver. 1 registered [ 0.838008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.841069] PTP clock support registered [ 0.843180] EDAC MC: Ver: 3.0.0 [ 0.844453] PCI: Using ACPI for IRQ routing [ 0.847059] NetLabel: Initializing [ 0.848007] NetLabel: domain hash size = 128 [ 0.849006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.851080] NetLabel: unlabeled traffic allowed by default [ 0.853156] vgaarb: loaded [ 0.854388] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.856015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.862892] clocksource: Switched to clocksource kvm-clock [ 0.970852] VFS: Disk quotas dquot_6.6.0 [ 0.972489] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.975044] *** VALIDATE ramfs *** [ 0.976376] *** VALIDATE hugetlbfs *** [ 0.977945] pnp: PnP ACPI init [ 0.980360] pnp: PnP ACPI: found 6 devices [ 0.998777] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.003171] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.005764] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.008012] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.010712] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.014150] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.017807] NET: Registered protocol family 2 [ 1.020766] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.026625] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.029878] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.034463] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.037765] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040820] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.043860] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046516] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049158] NET: Registered protocol family 1 [ 1.052311] RPC: Registered named UNIX socket transport module. [ 1.054382] RPC: Registered udp transport module. [ 1.055684] RPC: Registered tcp transport module. [ 1.057411] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.059763] NET: Registered protocol family 44 [ 1.061205] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.063019] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065073] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.067557] PCI: CLS 0 bytes, default 64 [ 1.069071] Unpacking initramfs... [ 2.582954] debug: unmapping init [mem 0xffff8ddc3cc54000-0xffff8ddc3ffbffff] [ 2.586648] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.588608] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.591255] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.181371] Initialise system trusted keyrings [ 3.183989] Key type blacklist registered [ 3.185910] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.197071] zbud: loaded [ 3.200257] *** VALIDATE nfs *** [ 3.201505] *** VALIDATE nfs4 *** [ 3.203674] pstore: using deflate compression [ 3.207974] Platform Keyring initialized [ 3.383540] NET: Registered protocol family 38 [ 3.385262] Key type asymmetric registered [ 3.386974] Asymmetric key parser 'x509' registered [ 3.388532] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.391971] io scheduler mq-deadline registered [ 3.393730] io scheduler kyber registered [ 3.395767] io scheduler bfq registered [ 3.397357] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.400703] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.403490] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.406803] ACPI: Power Button [PWRF] [ 3.507885] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.605618] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.796636] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.899291] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.109838] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.141684] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.173630] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.178993] Non-volatile memory driver v1.3 [ 4.180323] Linux agpgart interface v0.103 [ 4.210724] virtio_blk virtio1: [vda] 67976 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.214632] vda: detected capacity change from 0 to 34803712 [ 4.239318] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.246421] vdb: detected capacity change from 0 to 1073741824 [ 4.265548] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.268534] vdc: detected capacity change from 0 to 2621440000 [ 4.290216] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.292962] vdd: detected capacity change from 0 to 2621440000 [ 4.314451] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.319763] vde: detected capacity change from 0 to 4294967296 [ 4.337333] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.340117] vdf: detected capacity change from 0 to 4294967296 [ 4.346820] libphy: Fixed MDIO Bus: probed [ 4.356602] usbcore: registered new interface driver usbserial_generic [ 4.358760] usbserial: USB Serial support registered for generic [ 4.362144] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.367437] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.369513] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.375147] mousedev: PS/2 mouse device common for all mice [ 4.379913] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.383386] rtc_cmos 00:05: RTC can wake from S4 [ 4.402052] rtc_cmos 00:05: registered as rtc0 [ 4.403068] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.405706] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.406841] intel_pstate: CPU model not supported [ 4.418613] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.429657] hid: raw HID events driver (C) Jiri Kosina [ 4.431557] usbcore: registered new interface driver usbhid [ 4.433411] usbhid: USB HID core driver [ 4.434956] drop_monitor: Initializing network drop monitor service [ 4.437445] Initializing XFRM netlink socket [ 4.439483] NET: Registered protocol family 10 [ 4.443264] Segment Routing with IPv6 [ 4.444630] NET: Registered protocol family 17 [ 4.446860] mpls_gso: MPLS GSO support [ 4.451346] RAS: Correctable Errors collector initialized. [ 4.453494] AVX version of gcm_enc/dec engaged. [ 4.455234] AES CTR mode by8 optimization enabled [ 4.576814] sched_clock: Marking stable (4576739080, 0)->(5419528102, -842789022) [ 4.581210] registered taskstats version 1 [ 4.582901] Loading compiled-in X.509 certificates [ 4.584633] zswap: loaded using pool lzo/zbud [ 4.648173] Key type big_key registered [ 4.669361] Key type encrypted registered [ 4.674856] ima: No TPM chip found, activating TPM-bypass! [ 4.678695] ima: Allocated hash algorithm: sha1 [ 4.680574] ima: No architecture policies found [ 4.682629] evm: Initialising EVM extended attributes: [ 4.684750] evm: security.selinux [ 4.687171] evm: security.ima [ 4.688618] evm: security.capability [ 4.692461] evm: HMAC attrs: 0x1 [ 4.700251] rtc_cmos 00:05: setting system clock to 2025-10-24 12:49:51 UTC (1761310191) [ 4.709919] debug: unmapping init [mem 0xffffffff8ce03000-0xffffffff8cffffff] [ 4.716728] debug: unmapping init [mem 0xffffffff8bb82000-0xffffffff8be58fff] [ 4.723070] Write protecting the kernel read-only data: 28672k [ 4.729264] debug: unmapping init [mem 0xffffffff8a203000-0xffffffff8a3fffff] [ 4.732403] debug: unmapping init [mem 0xffffffff8ab14000-0xffffffff8abfffff] [ 4.796292] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.804569] systemd[1]: Detected virtualization kvm. [ 4.806409] systemd[1]: Detected architecture x86-64. [ 4.808034] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.839793] systemd[1]: No hostname configured. [ 4.841373] systemd[1]: Set hostname to . [ 4.844252] random: systemd: uninitialized urandom read (16 bytes read) [ 4.846370] systemd[1]: Initializing machine ID from random generator. [ 4.913954] random: ln: uninitialized urandom read (6 bytes read) [ 5.099267] random: systemd: uninitialized urandom read (16 bytes read) [ 5.102288] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.109346] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.117075] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.140897] device-mapper: uevent: version 1.0.3 [ 6.143900] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.692640] virtio_net virtio0 ens2: renamed from eth0 [ 7.754207] random: fast init done [ 8.087839] scsi host0: ata_piix [ 8.100136] scsi host1: ata_piix [ 8.101711] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.104466] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.207898] random: crng init done [ 13.209704] random: 7 urandom warning(s) missed due to ratelimiting [ 13.497946] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.430428] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.731514] printk: systemd: 19 output lines suppressed due to ratelimiting [ 16.202087] SELinux: Disabled at runtime. [ 16.290774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.300553] systemd[1]: Detected virtualization kvm. [ 16.302449] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.223908] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.227811] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.233209] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.237426] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.240408] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.247673] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.255248] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 17.564892] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.016243] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.655781] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.697395] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.302727] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.437159] EDAC sbridge: Ver: 1.1.2 [ 22.580092] Key type dns_resolver registered [ 22.979548] NFS: Registering the id_resolver key type [ 22.981290] Key type id_resolver registered [ 22.982796] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg408-server login: [ 45.337254] libcfs: loading out-of-tree module taints kernel. [ 45.351376] alg: No test for adler32 (adler32-zlib) [ 46.102286] Key type ._llcrypt registered [ 46.103135] Key type .llcrypt registered [ 46.133720] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 52.435874] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 52.906826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.115171] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 53.368875] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 53.371262] LNet: Accept secure, port 988 [ 54.967144] Key type lgssc registered [ 55.519157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.566646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 62.360377] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.374494] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.545612] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.890637] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.362789] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 78.390559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.415115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 79.496196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 79.505446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 79.540071] Lustre: lustre-MDT0000: new disk, initializing [ 79.562178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 79.568858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 80.927471] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 85.896827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.925808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.950068] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 85.958133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 85.959910] Lustre: Skipped 1 previous similar message [ 85.997421] Lustre: lustre-MDT0001: new disk, initializing [ 86.015852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 86.024346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 86.026953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.360271] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 92.602218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.628598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 92.631323] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 92.717197] Lustre: lustre-OST0000: new disk, initializing [ 92.719329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 92.737618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.546415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 93.548825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 94.184827] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 99.610861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.636706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 99.640553] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 99.677959] Lustre: lustre-OST0001: new disk, initializing [ 99.679598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 99.696373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.910456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 100.914244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 101.221771] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 106.655456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.210824] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 117.510945] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing check_logdir /tmp/testlogs/ [ 119.190607] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing yml_node [ 120.769362] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 121.592764] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 122.435938] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 122.914526] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:51:49 EDT 2025 [ 125.491610] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 125.958360] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 141.792164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 141.792500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.797784] Lustre: Skipped 3 previous similar messages [ 141.800615] Lustre: Skipped 2 previous similar messages [ 143.744793] Lustre: server umount lustre-MDT0000 complete [ 145.073591] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310331 with bad export cookie 2918211593029937847 [ 145.075116] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 145.076794] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 146.911698] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 146.911710] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.915629] LustreError: Skipped 1 previous similar message [ 146.916022] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 146.921044] Lustre: Skipped 1 previous similar message [ 150.943830] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 150.944584] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 150.945742] Lustre: Skipped 2 previous similar messages [ 150.949931] LustreError: Skipped 1 previous similar message [ 151.285023] Lustre: server umount lustre-MDT0001 complete [ 158.631166] Lustre: server umount lustre-OST0000 complete [ 174.047283] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 174.222704] Lustre: server umount lustre-OST0001 complete [ 178.144120] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 183.805185] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 190.943785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.967234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.427538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.824179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.438340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.465058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.565615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 202.578520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 202.619703] Lustre: lustre-MDT0000: new disk, initializing [ 202.658740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.668256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.407487] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 208.972896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.995837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.029572] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 209.128228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.508278] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 214.669573] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 216.657650] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 219.050506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.075299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.078732] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 219.171420] Lustre: lustre-OST0000: new disk, initializing [ 219.173483] Lustre: Skipped 1 previous similar message [ 219.176456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 219.179315] Lustre: Skipped 2 previous similar messages [ 219.208675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.211694] Lustre: Skipped 1 previous similar message [ 220.600407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.604037] Lustre: Skipped 1 previous similar message [ 220.606515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.782773] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 224.903426] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 226.500192] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 226.581727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 228.390073] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 228.490507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 230.881522] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 230.881978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.885108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 230.889356] Lustre: Skipped 1 previous similar message [ 230.893764] Lustre: Skipped 1 previous similar message [ 235.474957] Lustre: server umount lustre-OST0000 complete [ 239.008723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 239.015581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 239.018134] Lustre: Skipped 1 previous similar message [ 251.359190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 251.360592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.365878] Lustre: Skipped 7 previous similar messages [ 251.406029] Lustre: server umount lustre-MDT0000 complete [ 253.089060] LustreError: 14304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310439 with bad export cookie 2918211593029939289 [ 253.089722] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 253.093729] LustreError: 14304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 254.852690] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:54:01 (1761310441) [ 257.705492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.932533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.965631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.745494] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 263.136668] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.251298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 265.123849] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 267.747304] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 269.862675] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 272.286920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.124581] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 276.819812] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 283.619165] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 283.619337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.619900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 283.623088] LustreError: Skipped 2 previous similar messages [ 283.628122] Lustre: Skipped 2 previous similar messages [ 285.261973] Lustre: server umount lustre-OST0000 complete [ 285.263780] Lustre: Skipped 1 previous similar message [ 288.738358] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.743219] LustreError: Skipped 1 previous similar message [ 290.309911] LustreError: 18694:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310477 with bad export cookie 2918211593029940143 [ 290.312208] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.315813] LustreError: 18694:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 296.251210] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 297.470479] Key type lgssc unregistered [ 297.606558] LNet: 22362:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.610053] LNet: Removed LNI 192.168.204.108@tcp [ 298.035647] Key type .llcrypt unregistered [ 298.037348] Key type ._llcrypt unregistered [ 307.900333] alg: No test for adler32 (adler32-zlib) [ 308.652451] Key type ._llcrypt registered [ 308.653939] Key type .llcrypt registered [ 308.704430] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 309.134548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.368188] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 309.463632] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 309.466935] LNet: Accept secure, port 988 [ 311.074404] Key type lgssc registered [ 311.528894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 314.811635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 316.019424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.049251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 317.949676] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 321.504492] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.880952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.035384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 323.787868] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 326.721218] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 328.903457] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 331.564937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 331.694158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.452582] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 336.533419] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 347.048085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 352.224610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.225565] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 352.230380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 352.232204] LustreError: Skipped 1 previous similar message [ 352.233796] Lustre: Skipped 1 previous similar message [ 353.111449] Lustre: server umount lustre-OST0000 complete [ 356.480736] Lustre: server umount lustre-MDT0000 complete [ 357.344247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.346031] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.349952] Lustre: Skipped 2 previous similar messages [ 357.785263] LustreError: 23802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310544 with bad export cookie 9780197045288323183 [ 357.788047] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.790254] LustreError: 23802:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.909176] Lustre: server umount lustre-MDT0001 complete [ 362.650443] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 363.678519] Key type lgssc unregistered [ 363.788262] LNet: 27846:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.790947] LNet: Removed LNI 192.168.204.108@tcp [ 364.087178] Key type .llcrypt unregistered [ 364.088178] Key type ._llcrypt unregistered [ 372.995130] alg: No test for adler32 (adler32-zlib) [ 373.757754] Key type ._llcrypt registered [ 373.758798] Key type .llcrypt registered [ 373.823706] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 374.427123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.593153] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 374.685882] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 374.688362] LNet: Accept secure, port 988 [ 376.279124] Key type lgssc registered [ 376.658713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 379.460285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.597536] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.610058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.138191] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 385.161033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.302985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.854807] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 389.525587] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 391.455833] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 396.334054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 396.474296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.479251] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 398.177326] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 400.984053] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 405.986424] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.986537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.987240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 405.996587] Lustre: Skipped 1 previous similar message [ 409.444237] Lustre: server umount lustre-OST0000 complete [ 411.104103] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 411.106105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.108654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.115213] Lustre: Skipped 1 previous similar message [ 411.117493] Lustre: Skipped 1 previous similar message [ 416.224948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 416.227786] Lustre: Skipped 2 previous similar messages [ 417.224925] Lustre: server umount lustre-MDT0000 complete [ 418.627169] LustreError: 29286:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310605 with bad export cookie 15521700207012567389 [ 418.628767] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 418.631621] LustreError: 29286:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 418.764965] Lustre: server umount lustre-MDT0001 complete [ 424.005732] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 425.231227] Key type lgssc unregistered [ 425.364738] LNet: 33302:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.368741] LNet: Removed LNI 192.168.204.108@tcp [ 425.706871] Key type .llcrypt unregistered [ 425.707855] Key type ._llcrypt unregistered [ 435.843420] alg: No test for adler32 (adler32-zlib) [ 436.595538] Key type ._llcrypt registered [ 436.597109] Key type .llcrypt registered [ 436.643149] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 437.065804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 437.412197] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 437.517454] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 437.520220] LNet: Accept secure, port 988 [ 439.119107] Key type lgssc registered [ 439.616077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 442.837752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.994296] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.008329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.545076] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 448.593246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.710810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.254088] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 452.752459] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.765567] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 456.986617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.097631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.578744] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 460.196305] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 461.163804] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 470.496650] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 470.499699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.505372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.569048] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.569337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.572578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.647362] Lustre: server umount lustre-OST0000 complete [ 478.688639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.694545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 478.696995] Lustre: Skipped 2 previous similar messages [ 482.179259] Lustre: server umount lustre-MDT0000 complete [ 483.501118] LustreError: 34743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310670 with bad export cookie 17314878487844788817 [ 483.503238] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.504976] LustreError: 34743:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 483.634924] Lustre: server umount lustre-MDT0001 complete [ 488.656114] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 489.845532] Key type lgssc unregistered [ 489.976368] LNet: 38412:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.980455] LNet: Removed LNI 192.168.204.108@tcp [ 490.314437] Key type .llcrypt unregistered [ 490.315977] Key type ._llcrypt unregistered [ 499.948817] alg: No test for adler32 (adler32-zlib) [ 500.701269] Key type ._llcrypt registered [ 500.702278] Key type .llcrypt registered [ 500.736628] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 501.054409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 501.279710] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 501.360304] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 501.362343] LNet: Accept secure, port 988 [ 502.935152] Key type lgssc registered [ 503.273094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 505.975762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.131548] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.147185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.626948] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 511.611197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.737996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 513.239783] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 515.995880] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 517.912947] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 520.354362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.502869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.142745] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 522.535909] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 525.103701] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 532.959719] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.963226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.969417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.594196] Lustre: lustre-OST0000: Not available for connect from 192.168.204.8@tcp (stopping) [ 537.057619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.058166] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 538.286389] Lustre: server umount lustre-OST0000 complete [ 551.658551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.663326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.665569] Lustre: Skipped 3 previous similar messages [ 557.838953] Lustre: server umount lustre-MDT0000 complete [ 558.048387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.053800] LustreError: Skipped 1 previous similar message [ 559.212108] LustreError: 41822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310746 with bad export cookie 9748023126903453554 [ 559.215511] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.216671] LustreError: 41822:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 565.450895] Lustre: server umount lustre-MDT0001 complete [ 570.242356] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 571.421681] Key type lgssc unregistered [ 571.557461] LNet: 43700:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.560591] LNet: Removed LNI 192.168.204.108@tcp [ 571.896222] Key type .llcrypt unregistered [ 571.897403] Key type ._llcrypt unregistered [ 580.968430] alg: No test for adler32 (adler32-zlib) [ 581.720397] Key type ._llcrypt registered [ 581.722013] Key type .llcrypt registered [ 581.756228] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 582.145054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 582.447968] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 582.540259] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 582.542062] LNet: Accept secure, port 988 [ 584.119118] Key type lgssc registered [ 584.506915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.303568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.445734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.460983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.883385] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 592.711637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.832254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 594.089745] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 596.399796] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 598.158755] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 600.290728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.401776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.717552] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 603.492025] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 604.205206] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 612.832855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.836042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.838170] Lustre: Skipped 2 previous similar messages [ 612.840436] Lustre: Skipped 1 previous similar message [ 615.968201] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.8@tcp (stopping) [ 617.939326] Lustre: server umount lustre-MDT0000 complete [ 617.963577] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.990941] LustreError: Skipped 2 previous similar messages [ 621.037694] LustreError: 45138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310807 with bad export cookie 7348512884709674556 [ 621.040617] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.055201] LustreError: 45138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 621.069164] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.108330] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 623.073292] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.078755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.113918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 627.111354] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.123029] LustreError: Skipped 2 previous similar messages [ 628.297737] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.8@tcp (stopping) [ 628.304092] Lustre: Skipped 3 previous similar messages [ 635.360207] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 635.655545] Lustre: server umount lustre-MDT0001 complete [ 649.955677] Lustre: server umount lustre-OST0000 complete [ 663.288691] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 665.152328] Key type lgssc unregistered [ 665.339556] LNet: 49138:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.343859] LNet: Removed LNI 192.168.204.108@tcp [ 665.762825] Key type .llcrypt unregistered [ 665.764762] Key type ._llcrypt unregistered [ 677.909721] alg: No test for adler32 (adler32-zlib) [ 678.661530] Key type ._llcrypt registered [ 678.663109] Key type .llcrypt registered [ 678.725146] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 679.183481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.353935] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 679.445592] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 679.447921] LNet: Accept secure, port 988 [ 681.031125] Key type lgssc registered [ 681.403278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.290751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 701.791418] LustreError: 50588:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 732.511189] LustreError: 50588:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 762.208818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.155217] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 767.764061] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 805.904626] Lustre: server umount lustre-OST0000 complete [ 813.196614] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 814.206578] Key type lgssc unregistered [ 814.320379] LNet: 52254:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.323813] LNet: Removed LNI 192.168.204.108@tcp [ 814.615470] Key type .llcrypt unregistered [ 814.616610] Key type ._llcrypt unregistered [ 823.420648] alg: No test for adler32 (adler32-zlib) [ 824.172372] Key type ._llcrypt registered [ 824.173967] Key type .llcrypt registered [ 824.220864] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 824.682702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.851266] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 824.942954] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 824.945714] LNet: Accept secure, port 988 [ 826.543123] Key type lgssc registered [ 826.936831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 829.728105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.908968] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.934430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.579967] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 835.968374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.064708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 836.120669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.651786] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 840.303770] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 842.229665] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 844.497350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 844.616192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.668476] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 846.054794] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 848.663524] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 850.785704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.785765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 850.793261] Lustre: Skipped 1 previous similar message [ 851.561523] Lustre: server umount lustre-OST0000 complete [ 856.032167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.036369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.038381] Lustre: Skipped 2 previous similar messages [ 861.151884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.154256] Lustre: Skipped 1 previous similar message [ 866.271779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.273770] Lustre: Skipped 1 previous similar message [ 867.295106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 867.330696] Lustre: server umount lustre-MDT0000 complete [ 868.577972] LustreError: 53697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311055 with bad export cookie 10195964056072688530 [ 868.580256] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.582540] LustreError: 53697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 868.690046] Lustre: server umount lustre-MDT0001 complete [ 873.303203] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 874.368570] Key type lgssc unregistered [ 874.492410] LNet: 57364:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.495491] LNet: Removed LNI 192.168.204.108@tcp [ 874.794850] Key type .llcrypt unregistered [ 874.796318] Key type ._llcrypt unregistered [ 883.555504] alg: No test for adler32 (adler32-zlib) [ 884.307335] Key type ._llcrypt registered [ 884.308446] Key type .llcrypt registered [ 884.344642] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 884.768461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.920318] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 884.997328] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 884.999062] LNet: Accept secure, port 988 [ 886.575109] Key type lgssc registered [ 886.932713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 889.405290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.911228] LustreError: 58798:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 937.631209] LustreError: 58798:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 967.327162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.883030] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 971.498467] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.758197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.949398] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.972450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.992048] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 991.283431] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 993.911745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.025080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.412767] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 997.672702] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 999.436887] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1000.415751] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.419399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.424146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.513811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.513874] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.517864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.170138] Lustre: server umount lustre-OST0000 complete [ 1020.880448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.881187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.885323] Lustre: Skipped 1 previous similar message [ 1020.887134] Lustre: Skipped 1 previous similar message [ 1026.015845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.017572] Lustre: Skipped 1 previous similar message [ 1026.564976] Lustre: server umount lustre-MDT0000 complete [ 1027.758863] LustreError: 58805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311214 with bad export cookie 8824211516594462522 [ 1027.761126] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.761322] LustreError: 58805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1041.887114] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1041.975576] Lustre: server umount lustre-MDT0001 complete [ 1046.374037] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1047.338350] Key type lgssc unregistered [ 1047.463493] LNet: 62668:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.467163] LNet: Removed LNI 192.168.204.108@tcp [ 1047.779803] Key type .llcrypt unregistered [ 1047.781242] Key type ._llcrypt unregistered [ 1056.258396] alg: No test for adler32 (adler32-zlib) [ 1057.010371] Key type ._llcrypt registered [ 1057.012071] Key type .llcrypt registered [ 1057.045483] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1057.336674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.542349] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1057.613612] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1057.616032] LNet: Accept secure, port 988 [ 1059.191099] Key type lgssc registered [ 1059.521449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.008518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.129912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.143873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.417291] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1067.132257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.247256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.638762] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1071.056415] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1072.912290] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.161673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.269396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.701554] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1078.372507] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1079.067310] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1093.089747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.091179] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.091231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.094024] Lustre: Skipped 1 previous similar message [ 1097.556934] Lustre: server umount lustre-OST0000 complete [ 1103.330599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.335510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.336693] Lustre: Skipped 1 previous similar message [ 1103.336936] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1103.340447] Lustre: Skipped 2 previous similar messages [ 1105.007885] Lustre: server umount lustre-MDT0000 complete [ 1108.200022] LustreError: 64107:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311294 with bad export cookie 9893458622376766620 [ 1108.207335] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.219186] LustreError: 64107:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1108.538702] Lustre: server umount lustre-MDT0001 complete [ 1117.788322] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1120.142591] Key type lgssc unregistered [ 1120.412589] LNet: 67799:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.428330] LNet: Removed LNI 192.168.204.108@tcp [ 1121.110971] Key type .llcrypt unregistered [ 1121.112541] Key type ._llcrypt unregistered [ 1146.934212] hrtimer: interrupt took 5200730 ns [ 1149.614490] alg: No test for adler32 (adler32-zlib) [ 1150.379899] Key type ._llcrypt registered [ 1150.381282] Key type .llcrypt registered [ 1150.563215] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1152.319775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1152.913943] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1153.345355] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1153.358729] LNet: Accept secure, port 988 [ 1155.100022] Key type lgssc registered [ 1156.653751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1166.225830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.759871] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.814551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.328281] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1172.971678] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.740565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.082233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.860757] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1185.351797] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1188.787668] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1192.785276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.989298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.159093] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1199.143417] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 1200.584140] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.784156] Lustre: Failing over lustre-MDT0000 [ 1204.921628] Lustre: server umount lustre-MDT0000 complete [ 1207.318283] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.803922] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1208.807759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.314277] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.328952] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.335395] LustreError: Skipped 3 previous similar messages [ 1210.029794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.104737] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.109714] LustreError: 11-0: MGC192.168.204.108@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1210.118609] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0x2ed7cd1ad6878ad2 to 0x2ed7cd1ad6878ebb [ 1210.127336] Lustre: MGC192.168.204.108@tcp: Connection restored to (at 0@lo) [ 1210.216117] Lustre: *** cfs_fail_loc=135, val=0*** [ 1210.277177] BUG: spinlock bad magic on CPU#3, llog_process_th/72711 [ 1210.282858] lock: 0xffff8ddc855378c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1210.291450] CPU: 3 PID: 72711 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1210.295441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1210.298430] Call Trace: [ 1210.300127] ? dump_stack+0xbb/0x10e [ 1210.301371] ? spin_dump.cold.3+0x32/0x65 [ 1210.302412] ? do_raw_spin_lock+0xce/0x190 [ 1210.303492] ? _raw_spin_lock+0x21/0x30 [ 1210.304456] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1210.305724] ? mdd_process_config+0x368/0x9c0 [mdd] [ 1210.312017] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 1210.313731] ? mdt_init0+0x650/0x1750 [mdt] [ 1210.315171] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1210.316773] ? obd_setup+0x196/0x3e0 [obdclass] [ 1210.318806] ? class_setup+0x6ee/0x9e0 [obdclass] [ 1210.320650] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 1210.322652] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 1210.324914] ? llog_process_thread+0x109e/0x22a0 [obdclass] [ 1210.327181] ? keys_fill+0x10d/0x1b0 [obdclass] [ 1210.328809] ? lu_context_init+0xc8/0x240 [obdclass] [ 1210.330739] ? llog_process+0x40/0x40 [obdclass] [ 1210.332276] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 1210.334312] ? kthread+0x1d1/0x200 [ 1210.335413] ? set_kthread_struct+0x70/0x70 [ 1210.336797] ? ret_from_fork+0x1f/0x30 [ 1210.340447] LustreError: 72711:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 1210.344158] LustreError: 72711:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -2 [ 1210.348385] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1210.348385] [ 1210.354819] LustreError: 15c-8: MGC192.168.204.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1210.385538] LustreError: 72699:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1210.396311] LustreError: 72699:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1210.409693] LustreError: 72699:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1210.416171] LustreError: 69536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311397 with bad export cookie 3375391960889527995 [ 1210.434064] Lustre: server umount lustre-MDT0000 complete [ 1210.439908] LustreError: 72699:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1214.529789] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.536564] LustreError: Skipped 4 previous similar messages [ 1216.684601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.636321] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0x2ed7cd1ad6878ebb to 0x2ed7cd1ad6878f4e [ 1222.647195] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.686646] Lustre: MGC192.168.204.108@tcp: Connection restored to (at 0@lo) [ 1222.718677] LustreError: Skipped 13 previous similar messages [ 1223.145826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.227052] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1225.743131] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1228.268085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1228.301294] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1228.364311] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 1228.370643] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1238.504080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.527239] Lustre: Skipped 1 previous similar message [ 1238.543682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.548909] Lustre: Skipped 1 previous similar message [ 1243.671824] Lustre: server umount lustre-OST0000 complete [ 1248.738925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.741173] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1248.746108] Lustre: Skipped 1 previous similar message [ 1248.746872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.773275] Lustre: server umount lustre-MDT0000 complete [ 1253.863726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.881815] LustreError: Skipped 1 previous similar message [ 1256.573385] LustreError: 69537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311443 with bad export cookie 3375391960889528142 [ 1256.576544] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.585645] LustreError: 69537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1256.591526] LustreError: Skipped 1 previous similar message [ 1265.107257] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1266.881404] Key type lgssc unregistered [ 1267.072750] LNet: 74597:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.079359] LNet: Removed LNI 192.168.204.108@tcp [ 1267.598171] Key type .llcrypt unregistered [ 1267.600673] Key type ._llcrypt unregistered [ 1282.419046] alg: No test for adler32 (adler32-zlib) [ 1283.168390] Key type ._llcrypt registered [ 1283.171181] Key type .llcrypt registered [ 1283.265876] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1283.945439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1284.196398] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1284.332457] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1284.334739] LNet: Accept secure, port 988 [ 1285.951138] Key type lgssc registered [ 1286.768530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.392447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.621539] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.656135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.485335] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1298.917505] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.153104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.360527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.783347] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1313.178653] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.235677] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1323.634649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.899988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.914361] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 1326.825222] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1332.212086] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1340.356371] Lustre: Failing over lustre-MDT0000 [ 1342.565761] Lustre: server umount lustre-MDT0000 complete [ 1344.029548] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.483576] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1344.485962] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.499719] Lustre: Skipped 2 previous similar messages [ 1349.144709] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.150368] LustreError: Skipped 3 previous similar messages [ 1351.437411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.519467] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1351.520533] LustreError: 11-0: MGC192.168.204.108@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1351.526995] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0x6769f9e8e6f1a114 to 0x6769f9e8e6f1a4f6 [ 1351.546707] Lustre: MGC192.168.204.108@tcp: Connection restored to 192.168.204.108@tcp (at 0@lo) [ 1351.632118] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1351.699202] LustreError: 79229:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 1351.707797] LustreError: 79229:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -9 [ 1351.719543] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1351.719543] [ 1351.747906] LustreError: 15c-8: MGC192.168.204.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1351.757829] LustreError: 79217:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1351.762081] LustreError: 79217:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 1351.766720] LustreError: 79217:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1351.781199] Lustre: server umount lustre-MDT0000 complete [ 1351.782769] LustreError: 79217:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1355.287382] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.309573] LustreError: Skipped 7 previous similar messages [ 1358.227537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.369263] LustreError: 11-0: MGC192.168.204.108@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1358.373233] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.397508] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0x6769f9e8e6f1a4f6 to 0x6769f9e8e6f1a590 [ 1358.412911] Lustre: MGC192.168.204.108@tcp: Connection restored to 192.168.204.108@tcp (at 0@lo) [ 1358.591082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.637393] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1361.427752] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1362.908230] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1363.944697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1363.967819] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1364.024911] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 1369.568306] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.575169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.588740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.177640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.180382] Lustre: Skipped 1 previous similar message [ 1375.426445] Lustre: server umount lustre-OST0000 complete [ 1379.296413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.302765] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.305952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.305960] Lustre: Skipped 1 previous similar message [ 1379.346994] Lustre: Skipped 2 previous similar messages [ 1379.397454] LustreError: Skipped 1 previous similar message [ 1384.420569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.432647] Lustre: Skipped 2 previous similar messages [ 1384.716581] Lustre: server umount lustre-MDT0000 complete [ 1388.365097] LustreError: 79232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311575 with bad export cookie 7451761837163521424 [ 1388.369102] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.373941] LustreError: 79232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1398.979708] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1401.299676] Key type lgssc unregistered [ 1401.528195] LNet: 81116:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.539132] LNet: Removed LNI 192.168.204.108@tcp [ 1402.163530] Key type .llcrypt unregistered [ 1402.168317] Key type ._llcrypt unregistered [ 1422.615264] alg: No test for adler32 (adler32-zlib) [ 1423.371459] Key type ._llcrypt registered [ 1423.373291] Key type .llcrypt registered [ 1423.497408] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1424.409935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.169881] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1425.407610] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1425.416504] LNet: Accept secure, port 988 [ 1427.129865] Key type lgssc registered [ 1428.510023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.913923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.557613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.622844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.641776] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1443.812395] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.945567] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.681695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.902771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.962756] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1461.225408] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.383586] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1472.355308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.711074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.417227] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1482.934472] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1489.130300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 1502.689322] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1502.701726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.730861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.737526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.739561] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1504.744051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.997312] Lustre: server umount lustre-OST0000 complete [ 1512.415915] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1512.424134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.443227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.445978] Lustre: Skipped 1 previous similar message [ 1514.976623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.976703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.538838] Lustre: server umount lustre-MDT0000 complete [ 1519.925877] LustreError: 82558:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311706 with bad export cookie 11458847441044930737 [ 1519.930858] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1519.936975] LustreError: 82558:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1520.214391] Lustre: server umount lustre-MDT0001 complete [ 1530.279406] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1532.868739] Key type lgssc unregistered [ 1533.168496] LNet: 86257:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.176173] LNet: Removed LNI 192.168.204.108@tcp [ 1533.929773] Key type .llcrypt unregistered [ 1533.932675] Key type ._llcrypt unregistered [ 1555.772183] alg: No test for adler32 (adler32-zlib) [ 1556.524409] Key type ._llcrypt registered [ 1556.528533] Key type .llcrypt registered [ 1556.655180] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1557.897214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1558.302835] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1558.522110] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1558.531726] LNet: Accept secure, port 988 [ 1560.207931] Key type lgssc registered [ 1561.337878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1568.489250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.970836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.050518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.095065] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1575.394534] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.515272] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.684252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.052054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1586.548702] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1593.342947] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1598.303966] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1604.436974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.754786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.144717] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1616.327719] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.182455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 1626.596488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.597031] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1626.617630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.670849] Lustre: server umount lustre-OST0000 complete [ 1631.715102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.715628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.724977] Lustre: Skipped 2 previous similar messages [ 1631.727665] Lustre: Skipped 1 previous similar message [ 1636.840433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.843371] Lustre: Skipped 2 previous similar messages [ 1636.968894] Lustre: server umount lustre-MDT0000 complete [ 1640.283903] LustreError: 87701:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311827 with bad export cookie 6793226665582385075 [ 1640.288391] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.295040] LustreError: 87701:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1640.600477] Lustre: server umount lustre-MDT0001 complete [ 1650.750767] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1652.983335] Key type lgssc unregistered [ 1653.246692] LNet: 91376:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.262532] LNet: Removed LNI 192.168.204.108@tcp [ 1654.135380] Key type .llcrypt unregistered [ 1654.139538] Key type ._llcrypt unregistered [ 1673.540277] alg: No test for adler32 (adler32-zlib) [ 1674.295529] Key type ._llcrypt registered [ 1674.297241] Key type .llcrypt registered [ 1674.408167] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1675.283733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1675.855675] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1676.020888] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1676.026586] LNet: Accept secure, port 988 [ 1677.703100] Key type lgssc registered [ 1678.706916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.933359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.317782] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.390725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.585061] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1692.644250] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.763148] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.976441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.192599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.854646] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1709.494936] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1715.095086] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1721.747562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1722.096725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.185744] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1734.152863] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1737.470171] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 1751.788498] Lustre: server umount lustre-OST0000 complete [ 1755.617308] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.622946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.645363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.180559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.184275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.209396] Lustre: Skipped 1 previous similar message [ 1759.519168] Lustre: 92153:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311939/real 1761311939] req@0000000014974399 x1846869347022080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761311946 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1759.545535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.556091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.566555] Lustre: Skipped 1 previous similar message [ 1760.223338] Lustre: 92150:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311939/real 1761311939] req@00000000df3791b0 x1846869347022208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761311946 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1761.759257] Lustre: 95706:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311942/real 1761311942] req@00000000a52d7001 x1846869347022912/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761311948 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1762.077795] Lustre: server umount lustre-MDT0000 complete [ 1764.321281] Lustre: 92150:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311944/real 1761311944] req@00000000561f3373 x1846869347023232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761311951 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1764.838692] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.864208] LustreError: Skipped 1 previous similar message [ 1765.768744] LustreError: 92821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311952 with bad export cookie 5933797535878622902 [ 1765.769806] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.780963] LustreError: 92821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1766.186509] Lustre: server umount lustre-MDT0001 complete [ 1778.164381] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 1781.245336] Key type lgssc unregistered [ 1781.524167] LNet: 96510:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.533778] LNet: Removed LNI 192.168.204.108@tcp [ 1782.252459] Key type .llcrypt unregistered [ 1782.254684] Key type ._llcrypt unregistered [ 1805.195837] alg: No test for adler32 (adler32-zlib) [ 1805.958610] Key type ._llcrypt registered [ 1805.963275] Key type .llcrypt registered [ 1806.099345] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 1807.062835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.482299] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 1807.693586] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 1807.697489] LNet: Accept secure, port 988 [ 1809.418282] Key type lgssc registered [ 1810.529211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.795544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.359459] LustreError: 97947:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 1866.079200] LustreError: 97947:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 1896.863215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.714161] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1906.329858] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1910.733337] Lustre: server umount lustre-OST0000 complete [ 1917.414881] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:21:43 (1761312103) [ 1922.027323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.274640] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.316636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.194640] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1927.649281] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.484687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.774988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.652361] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1940.275150] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.647540] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1949.600886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1949.876585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.943390] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 1953.352932] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 1960.228568] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1973.727863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.733245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.758408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.829625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.832852] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.836670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.060743] Lustre: server umount lustre-OST0000 complete [ 1982.944618] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.945787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.964527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.987416] Lustre: Skipped 1 previous similar message [ 1982.995955] Lustre: Skipped 2 previous similar messages [ 1988.065497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.072056] Lustre: Skipped 1 previous similar message [ 1993.185119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.279152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1997.423677] Lustre: server umount lustre-MDT0000 complete [ 1999.333800] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.351698] LustreError: Skipped 1 previous similar message [ 2000.833197] LustreError: 99528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312187 with bad export cookie 2164633618992848717 [ 2000.836029] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.841076] LustreError: 99528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2001.134338] Lustre: server umount lustre-MDT0001 complete [ 2011.644928] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2014.724170] Key type lgssc unregistered [ 2014.997091] LNet: 103328:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.017602] LNet: Removed LNI 192.168.204.108@tcp [ 2015.840775] Key type .llcrypt unregistered [ 2015.842232] Key type ._llcrypt unregistered [ 2037.179386] alg: No test for adler32 (adler32-zlib) [ 2037.936437] Key type ._llcrypt registered [ 2037.938957] Key type .llcrypt registered [ 2038.059173] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2038.864105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.459400] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2039.656581] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2039.659831] LNet: Accept secure, port 988 [ 2041.295683] Key type lgssc registered [ 2042.204785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2049.110375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.448533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.511462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.621427] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2055.650813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.770545] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.828448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.185917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.139805] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2073.136133] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2080.070827] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2087.107712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.416580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.709390] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2098.648079] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2100.726873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 2109.924556] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2109.930628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.946739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.944872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.947493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.947700] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.980612] Lustre: Skipped 1 previous similar message [ 2115.561079] Lustre: server umount lustre-OST0000 complete [ 2119.648665] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2119.652068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.666950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2125.734869] Lustre: server umount lustre-MDT0000 complete [ 2126.304342] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.312435] LustreError: Skipped 1 previous similar message [ 2128.849027] LustreError: 104771:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312315 with bad export cookie 16134730396655842693 [ 2128.851016] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.855354] LustreError: 104771:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2129.299163] Lustre: server umount lustre-MDT0001 complete [ 2139.811678] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2142.201497] Key type lgssc unregistered [ 2142.432234] LNet: 108445:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.444889] LNet: Removed LNI 192.168.204.108@tcp [ 2143.308284] Key type .llcrypt unregistered [ 2143.310386] Key type ._llcrypt unregistered [ 2157.584297] alg: No test for adler32 (adler32-zlib) [ 2158.343400] Key type ._llcrypt registered [ 2158.350090] Key type .llcrypt registered [ 2158.489195] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2159.622089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.143296] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2160.404818] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2160.415522] LNet: Accept secure, port 988 [ 2162.152081] Key type lgssc registered [ 2163.534877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2170.772485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.359329] LustreError: 109743:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 2218.079281] LustreError: 109743:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 2248.929404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.592557] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2258.162917] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2262.612595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.539455] LustreError: 15c-8: MGC192.168.204.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2271.548548] LustreError: 110727:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2271.554517] LustreError: 110727:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2271.563832] LustreError: 110727:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 2271.679667] Lustre: server umount lustre-MDT0000 complete [ 2271.681453] LustreError: 110727:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2282.978617] Lustre: 111158:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312463/real 1761312463] req@000000008a023a91 x1846869855569856/t0(0) o251->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312469 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2283.156671] Lustre: server umount lustre-OST0000 complete [ 2290.203621] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 2296.490881] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2305.406254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2311.100734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2315.820308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2320.961084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2326.600962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2326.653487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.841104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2326.866449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2326.925622] Lustre: lustre-MDT0000: new disk, initializing [ 2326.984196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.006144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2330.658115] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2340.047263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2340.116214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.216314] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2340.255404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2340.258279] Lustre: Skipped 1 previous similar message [ 2340.317467] Lustre: lustre-MDT0001: new disk, initializing [ 2340.411343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.450970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2340.462341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2344.093746] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2352.320423] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2357.093750] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2362.845490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.913847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.919802] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2363.087521] Lustre: lustre-OST0000: new disk, initializing [ 2363.101873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2363.196973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.171144] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2368.562957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2368.575334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2374.881337] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2378.829303] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2379.021528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2382.658791] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2382.868375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2384.351945] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.357121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.358013] LustreError: Skipped 1 previous similar message [ 2384.370335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.375369] Lustre: Skipped 1 previous similar message [ 2389.478028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2389.482603] Lustre: Skipped 1 previous similar message [ 2390.639374] Lustre: server umount lustre-OST0000 complete [ 2394.593186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.596255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.600282] Lustre: Skipped 2 previous similar messages [ 2394.604684] Lustre: Skipped 1 previous similar message [ 2399.718183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2399.724066] Lustre: Skipped 1 previous similar message [ 2400.836169] Lustre: server umount lustre-MDT0000 complete [ 2404.191795] LustreError: 117300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312590 with bad export cookie 15240764891116144601 [ 2404.200824] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.242817] LustreError: 117300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2411.555083] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:29:57 (1761312597) [ 2414.485684] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2428.489727] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 2435.222158] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2444.688276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2451.246194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2456.174874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2460.864469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2466.195318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2466.262462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.472372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2466.497951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2466.564656] Lustre: lustre-MDT0000: new disk, initializing [ 2466.636502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.656739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2469.740464] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2477.462881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2477.514299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.577418] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2477.607087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2477.611250] Lustre: Skipped 1 previous similar message [ 2477.656883] Lustre: lustre-MDT0001: new disk, initializing [ 2477.709063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2477.755559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2477.788420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2480.742450] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2487.462813] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.896282] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2495.578623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2495.619387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.625242] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2495.807514] Lustre: lustre-OST0000: new disk, initializing [ 2495.810431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2495.901965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.124893] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2501.698702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2501.717267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2506.247687] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2509.230371] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2509.393366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2512.652848] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2512.841963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2517.474442] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2517.475310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.481089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.491097] Lustre: Skipped 1 previous similar message [ 2520.564838] Lustre: server umount lustre-OST0000 complete [ 2520.567770] Lustre: Skipped 1 previous similar message [ 2524.128980] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2524.132408] LustreError: Skipped 1 previous similar message [ 2524.145963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.713404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.719670] Lustre: Skipped 3 previous similar messages [ 2529.825451] Lustre: server umount lustre-MDT0000 complete [ 2532.835233] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.846758] LustreError: Skipped 1 previous similar message [ 2533.219303] LustreError: 121603:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312720 with bad export cookie 15240764891116145455 [ 2533.241294] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.788094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.153755] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.218489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.201821] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2545.632516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.166677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.743053] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2557.551850] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2561.380401] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2566.188889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.082341] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2574.059733] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.066283] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2586.080035] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2586.087856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.095825] Lustre: Skipped 1 previous similar message [ 2586.100133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.774650] Lustre: server umount lustre-OST0000 complete [ 2587.781848] Lustre: Skipped 1 previous similar message [ 2590.692827] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2595.306607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.309956] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.322934] Lustre: Skipped 1 previous similar message [ 2595.338886] LustreError: Skipped 1 previous similar message [ 2595.379426] LustreError: 125085:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312782 with bad export cookie 15240764891116146309 [ 2595.382526] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.392565] LustreError: 125085:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2601.439201] Lustre: 128252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312782/real 1761312782] req@00000000e8704ab3 x1846869855619008/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761312788 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2610.413339] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 2612.564542] Key type lgssc unregistered [ 2612.788754] LNet: 128856:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.796515] LNet: Removed LNI 192.168.204.108@tcp [ 2613.451381] Key type .llcrypt unregistered [ 2613.453844] Key type ._llcrypt unregistered [ 2627.465027] alg: No test for adler32 (adler32-zlib) [ 2628.218040] Key type ._llcrypt registered [ 2628.219451] Key type .llcrypt registered [ 2628.299722] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 2637.360489] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 2637.958800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.185789] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 2638.330780] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 2638.333616] LNet: Accept secure, port 988 [ 2639.951696] Key type lgssc registered [ 2640.672820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.623223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.626889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2655.648721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.314960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2664.489929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2664.546921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.800708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2665.826319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2665.870423] Lustre: lustre-MDT0000: new disk, initializing [ 2665.968266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.995391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2669.454249] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2677.512459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2677.598090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.674355] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2677.719885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2677.723590] Lustre: Skipped 1 previous similar message [ 2677.783879] Lustre: lustre-MDT0001: new disk, initializing [ 2677.876491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.918128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2677.929611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2681.550604] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2689.159619] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2693.135372] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2698.857484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2698.955092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.959808] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2699.291775] Lustre: lustre-OST0000: new disk, initializing [ 2699.305456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2699.431224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.634319] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2708.002233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2708.011198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2709.578039] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2712.591908] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2712.771993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2715.670901] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2715.842919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2718.179670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.181196] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.202400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2718.211123] Lustre: Skipped 1 previous similar message [ 2723.219760] Lustre: server umount lustre-OST0000 complete [ 2728.416598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.420609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.431550] Lustre: Skipped 2 previous similar messages [ 2728.458965] Lustre: Skipped 1 previous similar message [ 2733.536869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.541684] Lustre: Skipped 1 previous similar message [ 2739.680114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.682641] Lustre: Skipped 2 previous similar messages [ 2741.215130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2741.380992] Lustre: server umount lustre-MDT0000 complete [ 2744.800334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.827487] LustreError: Skipped 1 previous similar message [ 2744.964971] LustreError: 134220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312931 with bad export cookie 3076841036185047047 [ 2744.967564] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.977263] LustreError: 134220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.307296] Lustre: server umount lustre-MDT0001 complete [ 2751.431412] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:35:37 (1761312937) [ 2756.489499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.743447] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.804447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.239875] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2762.211916] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.768664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.977413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.008593] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2775.459716] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2779.630143] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.175948] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2782.178642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.192917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2787.269407] Lustre: server umount lustre-MDT0000 complete [ 2788.324714] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.333842] LustreError: Skipped 1 previous similar message [ 2789.866187] LustreError: 136660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312976 with bad export cookie 3076841036185047901 [ 2789.869221] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.883154] LustreError: 136660:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2795.667195] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:36:21 (1761312981) [ 2799.698115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.287553] LustreError: 139198:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 2827.554491] LustreError: 139198:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 2857.311319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.882356] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2867.005684] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2868.785359] Lustre: server umount lustre-OST0000 complete [ 2868.787526] Lustre: Skipped 1 previous similar message [ 2875.549987] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:37:41 (1761313061) [ 2881.363116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.687167] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.747712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.957975] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2892.174820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.260390] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.273979] LustreError: Skipped 1 previous similar message [ 2896.850554] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2903.384672] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2908.054138] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2913.094904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.771483] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2922.306162] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2937.316094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.318718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.325387] Lustre: Skipped 3 previous similar messages [ 2937.329120] Lustre: Skipped 3 previous similar messages [ 2939.061536] Lustre: server umount lustre-MDT0000 complete [ 2942.030180] LustreError: 143277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313128 with bad export cookie 3076841036185048566 [ 2942.032046] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.035287] LustreError: 143277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2950.623991] Lustre: 143679:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313131/real 1761313131] req@00000000d6a908b6 x1846870356828224/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313137 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2957.029400] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:39:02 (1761313142) [ 2963.013192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.285902] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.348831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.351656] Lustre: Skipped 2 previous similar messages [ 2967.297887] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2974.979738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.550442] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 2984.725523] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.973659] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2995.755567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3000.682276] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3007.870100] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3012.069873] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3013.555118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 3014.441514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3018.222896] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3018.450702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3023.842374] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3023.858645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.871765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3023.874227] Lustre: Skipped 1 previous similar message [ 3026.493674] Lustre: server umount lustre-OST0000 complete [ 3026.497977] Lustre: Skipped 2 previous similar messages [ 3031.519873] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3031.526658] LustreError: Skipped 1 previous similar message [ 3034.085555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.108332] Lustre: Skipped 2 previous similar messages [ 3039.201385] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.218042] LustreError: Skipped 3 previous similar messages [ 3041.630377] LustreError: 145138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313228 with bad export cookie 3076841036185049623 [ 3041.637530] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.645294] LustreError: 145138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3047.597929] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:40:33 (1761313233) [ 3052.003421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3069.536919] LustreError: 148690:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 3080.799459] LustreError: 148690:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 3109.535765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.542361] Lustre: Skipped 2 previous similar messages [ 3114.249884] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3120.786364] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3127.316163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.321300] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.334866] LustreError: Skipped 3 previous similar messages [ 3151.592380] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 3156.150995] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3163.037651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.930695] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3173.709046] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3179.210555] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3183.249781] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3183.448605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3187.535528] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3187.754825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3193.826828] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3193.831096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.849334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.851851] Lustre: Skipped 3 previous similar messages [ 3196.062323] Lustre: server umount lustre-MDT0000 complete [ 3196.073678] Lustre: Skipped 2 previous similar messages [ 3199.324668] LustreError: 148697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313386 with bad export cookie 3076841036185050666 [ 3199.346888] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3199.357748] LustreError: 148697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3213.160476] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:43:18 (1761313398) [ 3218.726718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3235.232599] LustreError: 153048:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 3246.495533] LustreError: 153048:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 3275.232484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.240765] Lustre: Skipped 2 previous similar messages [ 3279.237661] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3286.084354] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.306473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.935264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.948429] LustreError: Skipped 8 previous similar messages [ 3294.084266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 3298.032190] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3305.222589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.908236] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3314.683271] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.476064] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3325.004268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3325.085967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.089290] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3325.177410] Lustre: lustre-OST0001: new disk, initializing [ 3325.186158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3328.752854] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3331.093854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3331.103348] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3336.057274] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3339.544948] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3339.723686] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3343.308126] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3343.521869] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3345.379247] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3345.387972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.417930] Lustre: Skipped 3 previous similar messages [ 3345.424675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.428341] Lustre: Skipped 4 previous similar messages [ 3351.338283] Lustre: server umount lustre-OST0000 complete [ 3351.341735] Lustre: Skipped 2 previous similar messages [ 3366.885817] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3366.889568] LustreError: Skipped 3 previous similar messages [ 3379.167138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3381.907479] LustreError: 153054:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313568 with bad export cookie 3076841036185051604 [ 3381.910029] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.929767] LustreError: 153054:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3386.456403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.487521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3392.034042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.222988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.964678] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:46:24 (1761313584) [ 3400.089198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3401.194808] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3402.206103] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:46:28 (1761313588) [ 3406.297653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.486657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3406.515716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3409.857386] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3417.117113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.231363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3417.269543] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3421.492688] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3428.477344] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3433.302274] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3438.346528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3438.472895] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3441.388540] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3445.737817] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3448.335422] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3449.540377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3451.970182] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3452.133281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3453.921701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.928502] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3453.931773] Lustre: Skipped 5 previous similar messages [ 3472.840989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.095607] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3481.396437] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3485.012462] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3485.353310] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3485.366965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3485.368544] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.108@tcp (at 0@lo) [ 3485.378672] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 3485.390792] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 3485.392222] Lustre: Skipped 1 previous similar message [ 3486.212685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3489.501565] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3489.705414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3494.542576] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0cd45ca9000.ost_server_uuid 40 [ 3495.841735] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0cd45ca9000.ost_server_uuid in FULL state after 0 sec [ 3495.999572] LustreError: 159631:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 192.168.204.8@tcp) returned error from blocking AST (req@00000000e3ba342d x1846870356904256 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ac5abfea/0x2ab3229ee027aa1d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.8@tcp remote: 0x4e08919a2e65e320 expref: 6 pid: 161603 timeout: 3595 lvb_type: 0 [ 3496.027530] LustreError: 138-a: lustre-MDT0000: A client on nid 192.168.204.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3496.038074] LustreError: 159619:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.204.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ac5abfea/0x2ab3229ee027aa1d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.8@tcp remote: 0x4e08919a2e65e320 expref: 7 pid: 161603 timeout: 0 lvb_type: 0 [ 3501.026805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3501.035035] Lustre: Skipped 15 previous similar messages [ 3505.303265] Lustre: server umount lustre-OST0000 complete [ 3505.305658] Lustre: Skipped 4 previous similar messages [ 3517.547306] LustreError: 159614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313704 with bad export cookie 3076841036185052766 [ 3517.556510] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.574465] LustreError: 159614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3535.845461] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3538.205631] Key type lgssc unregistered [ 3538.494530] LNet: 166027:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.499342] LNet: Removed LNI 192.168.204.108@tcp [ 3539.263132] Key type .llcrypt unregistered [ 3539.266130] Key type ._llcrypt unregistered [ 3558.710959] alg: No test for adler32 (adler32-zlib) [ 3559.472458] Key type ._llcrypt registered [ 3559.474055] Key type .llcrypt registered [ 3559.604579] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3560.257514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3560.765260] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3560.951787] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 3560.960195] LNet: Accept secure, port 988 [ 3562.647071] Key type lgssc registered [ 3563.576889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3569.341600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.674136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.713184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.009460] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3575.778846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.706084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.006328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3584.511647] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3591.184948] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3596.717860] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3601.823201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3602.055787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.371146] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3611.609644] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3621.552336] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 3625.719304] Lustre: Failing over lustre-MDT0000 [ 3625.841308] Lustre: server umount lustre-MDT0000 complete [ 3626.978447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.979342] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3626.981512] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.985336] Lustre: Skipped 1 previous similar message [ 3632.100232] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.111322] LustreError: Skipped 4 previous similar messages [ 3634.143236] Lustre: 166800:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313813/real 1761313813] req@00000000961d19ee x1846871323586816/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313820 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3634.158390] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.265969] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.278756] LustreError: Skipped 5 previous similar messages [ 3642.477139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.521273] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.530547] LustreError: Skipped 5 previous similar messages [ 3650.535323] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0xf738d3edceb59b0d to 0xf738d3edceb59efd [ 3650.553267] Lustre: MGC192.168.204.108@tcp: Connection restored to 192.168.204.108@tcp (at 0@lo) [ 3650.846711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.893373] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3654.170611] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3655.834186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3655.840686] Lustre: lustre-MDT0000: Denying connection for new client 0dd4ad08-d4a4-487a-95ad-d318969e5cee (at 192.168.204.8@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3656.171576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3666.403414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.404784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3666.409921] Lustre: Skipped 2 previous similar messages [ 3666.417285] Lustre: Skipped 1 previous similar message [ 3669.622948] Lustre: server umount lustre-OST0000 complete [ 3672.055493] LustreError: 171368:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3672.060244] Lustre: 170663:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3672.067622] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 3672.095539] Lustre: 170663:0:(mdt_handler.c:7532:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 3676.640509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.650205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.334358] Lustre: server umount lustre-MDT0000 complete [ 3680.980634] LustreError: 168276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313867 with bad export cookie 17814221344392716029 [ 3680.983110] LustreError: 166799:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3680.986447] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.987208] LustreError: 168276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3687.444298] Lustre: server umount lustre-MDT0001 complete [ 3697.544537] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3699.709557] Key type lgssc unregistered [ 3699.921067] LNet: 172174:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.927448] LNet: Removed LNI 192.168.204.108@tcp [ 3700.399082] Key type .llcrypt unregistered [ 3700.400904] Key type ._llcrypt unregistered [ 3716.211344] alg: No test for adler32 (adler32-zlib) [ 3716.967384] Key type ._llcrypt registered [ 3716.969232] Key type .llcrypt registered [ 3717.041063] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3717.685618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3717.957218] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3718.104484] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 3718.107886] LNet: Accept secure, port 988 [ 3719.767908] Key type lgssc registered [ 3720.544093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.594457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.911944] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.959294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.846324] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3732.453286] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.133747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.490240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3739.352699] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3744.698645] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3749.694784] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3754.476770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3754.832291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.883097] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:225 [ 3758.858786] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3765.975577] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3772.391807] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3772.392064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.400723] Lustre: Skipped 1 previous similar message [ 3772.403411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3772.405354] Lustre: Skipped 1 previous similar message [ 3773.187338] Lustre: server umount lustre-OST0000 complete [ 3777.505966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.538662] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3777.541199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.581961] Lustre: Skipped 1 previous similar message [ 3782.624382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.627599] Lustre: Skipped 1 previous similar message [ 3783.355169] Lustre: server umount lustre-MDT0000 complete [ 3786.044277] LustreError: 173617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313972 with bad export cookie 1999757669996863679 [ 3786.060350] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3786.255967] Lustre: server umount lustre-MDT0001 complete [ 3796.335892] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3798.739327] Key type lgssc unregistered [ 3798.983624] LNet: 177288:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3798.994816] LNet: Removed LNI 192.168.204.108@tcp [ 3799.618717] Key type .llcrypt unregistered [ 3799.620740] Key type ._llcrypt unregistered [ 3825.411168] alg: No test for adler32 (adler32-zlib) [ 3826.165401] Key type ._llcrypt registered [ 3826.167724] Key type .llcrypt registered [ 3826.261339] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3827.088884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3827.360385] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3827.530395] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 3827.534901] LNet: Accept secure, port 988 [ 3829.199635] Key type lgssc registered [ 3830.060709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3835.909218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.199323] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.228754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.146932] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3842.532224] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.411361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.653100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3848.237829] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3853.581603] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3857.242867] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3861.496239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3861.845927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.004361] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3868.021067] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:257 [ 3870.212287] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3881.444407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.446218] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.456832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3881.461052] Lustre: Skipped 1 previous similar message [ 3885.471937] Lustre: server umount lustre-OST0000 complete [ 3891.681573] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3891.685833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.688637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.704942] Lustre: Skipped 2 previous similar messages [ 3894.251416] Lustre: server umount lustre-MDT0000 complete [ 3896.810580] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.824602] LustreError: Skipped 1 previous similar message [ 3897.252150] LustreError: 179018:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314084 with bad export cookie 13859811844108295597 [ 3897.253128] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3897.273821] LustreError: 179018:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3897.544282] Lustre: server umount lustre-MDT0001 complete [ 3905.770554] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3907.814946] Key type lgssc unregistered [ 3908.053804] LNet: 182691:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3908.060504] LNet: Removed LNI 192.168.204.108@tcp [ 3908.693738] Key type .llcrypt unregistered [ 3908.697305] Key type ._llcrypt unregistered [ 3923.812568] alg: No test for adler32 (adler32-zlib) [ 3924.565447] Key type ._llcrypt registered [ 3924.567705] Key type .llcrypt registered [ 3924.645552] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 3925.248490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3925.577335] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 3925.772626] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 3925.775114] LNet: Accept secure, port 988 [ 3927.399120] Key type lgssc registered [ 3928.179603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3936.489031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.913043] Lustre: *** cfs_fail_loc=135, val=0*** [ 3937.974474] LustreError: 184264:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 3937.977828] LustreError: 184264:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.108@tcp: cfg command failed: rc = -2 [ 3937.980932] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3937.980932] [ 3937.988211] LustreError: 15c-8: MGC192.168.204.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3938.019232] LustreError: 184225:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3938.024689] LustreError: 184225:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 3938.045388] LustreError: 184225:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 3943.391191] Lustre: 184225:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314124/real 1761314124] req@00000000abc5713b x1846871706305344/t0(0) o251->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314130 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3943.534649] Lustre: server umount lustre-MDT0000 complete [ 3943.537316] LustreError: 184225:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3948.926589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.256233] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.311698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.845660] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3954.659139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.952874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.239227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3962.134859] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 3968.158621] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3972.378325] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3974.626972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.627119] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3974.629280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.635211] Lustre: Skipped 1 previous similar message [ 3979.743938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.746775] Lustre: Skipped 2 previous similar messages [ 3979.803162] Lustre: server umount lustre-MDT0000 complete [ 3982.644360] LustreError: 184618:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314169 with bad export cookie 338432595900415278 [ 3982.646079] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3982.654395] LustreError: 184618:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3982.938301] Lustre: server umount lustre-MDT0001 complete [ 3990.338994] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 3992.612354] Key type lgssc unregistered [ 3992.986189] LNet: 187119:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3993.007271] LNet: Removed LNI 192.168.204.108@tcp [ 3993.674920] Key type .llcrypt unregistered [ 3993.678253] Key type ._llcrypt unregistered [ 4009.935316] alg: No test for adler32 (adler32-zlib) [ 4010.696504] Key type ._llcrypt registered [ 4010.698952] Key type .llcrypt registered [ 4010.795338] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4011.010941] Key type .llcrypt unregistered [ 4011.012495] Key type ._llcrypt unregistered [ 4021.653634] alg: No test for adler32 (adler32-zlib) [ 4022.407606] Key type ._llcrypt registered [ 4022.410939] Key type .llcrypt registered [ 4022.482352] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4023.370485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4023.655841] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4023.833545] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4023.838275] LNet: Accept secure, port 988 [ 4025.503184] Key type lgssc registered [ 4026.257383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4031.470752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.991327] LustreError: 189374:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 1: rc = -5 [ 4079.711308] LustreError: 189374:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.108@tcp: failed processing log, type 4: rc = -110 [ 4110.431206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.642872] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4119.104636] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4123.884316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.452662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.508417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4131.555781] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:289 [ 4134.210418] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4136.933277] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.512957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4139.835204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.284058] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4147.072791] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4150.458804] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4152.119084] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 4165.601308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.601846] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4165.617848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.734223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.742896] Lustre: Skipped 2 previous similar messages [ 4175.843425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4175.849678] Lustre: Skipped 1 previous similar message [ 4178.399286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4178.549850] Lustre: server umount lustre-OST0000 complete [ 4186.080616] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4186.081978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.094536] Lustre: Skipped 1 previous similar message [ 4186.098594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.672088] Lustre: server umount lustre-MDT0000 complete [ 4189.824856] LustreError: 189381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314376 with bad export cookie 14960222479753772341 [ 4189.827508] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4189.831345] LustreError: 189381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4189.974221] Lustre: server umount lustre-MDT0001 complete [ 4195.591134] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4197.285784] Key type lgssc unregistered [ 4197.480720] LNet: 193557:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4197.486776] LNet: Removed LNI 192.168.204.108@tcp [ 4198.018306] Key type .llcrypt unregistered [ 4198.023325] Key type ._llcrypt unregistered [ 4213.814299] alg: No test for adler32 (adler32-zlib) [ 4214.569446] Key type ._llcrypt registered [ 4214.572151] Key type .llcrypt registered [ 4214.659342] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4215.385929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4215.587252] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4215.749589] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4215.752974] LNet: Accept secure, port 988 [ 4217.391169] Key type lgssc registered [ 4218.136473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4223.435449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.694479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.720799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.299230] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4230.114322] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.169351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.413852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.226526] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4240.923774] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4244.688423] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4249.331458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.524749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.393614] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4254.630531] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:321 [ 4257.466658] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4261.142318] Lustre: Failing over lustre-MDT0000 [ 4261.289144] Lustre: server umount lustre-MDT0000 complete [ 4263.844607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.846654] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.859208] Lustre: Skipped 2 previous similar messages [ 4263.869232] LustreError: Skipped 1 previous similar message [ 4269.024291] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.032394] LustreError: Skipped 3 previous similar messages [ 4271.071196] Lustre: 194331:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314450/real 1761314450] req@0000000048d3502b x1846872010401536/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314457 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4271.089346] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.193478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.208090] LustreError: Skipped 7 previous similar messages [ 4278.530989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.481593] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.498080] LustreError: Skipped 8 previous similar messages [ 4288.483798] Lustre: Evicted from MGS (at 192.168.204.108@tcp) after server handle changed from 0x5cc4290fd22c55a6 to 0x5cc4290fd22c5988 [ 4288.494986] Lustre: MGC192.168.204.108@tcp: Connection restored to 192.168.204.108@tcp (at 0@lo) [ 4288.666268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.707285] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4291.785115] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4292.626133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4294.118748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4294.131643] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4294.154963] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:353 [ 4294.736727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 4302.493477] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 4314.088917] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4314.094937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.107452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.332736] Lustre: server umount lustre-OST0000 complete [ 4319.715102] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4319.718518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.724665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.735226] Lustre: Skipped 2 previous similar messages [ 4319.743563] Lustre: Skipped 1 previous similar message [ 4324.833597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.838418] Lustre: Skipped 2 previous similar messages [ 4325.058422] Lustre: server umount lustre-MDT0000 complete [ 4327.738077] LustreError: 194999:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314514 with bad export cookie 6684512894852028808 [ 4327.739529] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.743867] LustreError: 194999:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4328.049302] Lustre: server umount lustre-MDT0001 complete [ 4336.552931] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4338.735526] Key type lgssc unregistered [ 4338.981527] LNet: 200188:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4338.990217] LNet: Removed LNI 192.168.204.108@tcp [ 4339.532894] Key type .llcrypt unregistered [ 4339.535976] Key type ._llcrypt unregistered [ 4355.191710] alg: No test for adler32 (adler32-zlib) [ 4355.948802] Key type ._llcrypt registered [ 4355.951893] Key type .llcrypt registered [ 4356.073874] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4356.763493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4357.141504] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4357.321777] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4357.324783] LNet: Accept secure, port 988 [ 4358.993419] Key type lgssc registered [ 4359.820264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4366.246377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.841866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.909590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.474255] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4372.961498] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.082667] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.203473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.458147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.709195] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4389.084454] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4393.897726] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4399.269186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4399.631369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.922562] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4408.503339] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4418.099305] Lustre: lustre-OST0000: deleting orphan objects from 0x0:355 to 0x0:385 [ 4421.292828] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 4428.811505] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 4438.895053] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 4453.859640] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4453.863036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.864085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.876043] Lustre: Skipped 1 previous similar message [ 4457.152052] Lustre: server umount lustre-OST0000 complete [ 4461.537132] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4461.542468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.560569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.565809] Lustre: Skipped 1 previous similar message [ 4464.099607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.100281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4464.110768] Lustre: Skipped 1 previous similar message [ 4465.834577] Lustre: server umount lustre-MDT0000 complete [ 4468.462253] LustreError: 202437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314655 with bad export cookie 260597141842153611 [ 4468.465162] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.476517] LustreError: 202437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4468.778092] Lustre: server umount lustre-MDT0001 complete [ 4477.566423] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4479.348264] Key type lgssc unregistered [ 4479.541747] LNet: 205514:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4479.549223] LNet: Removed LNI 192.168.204.108@tcp [ 4480.103456] Key type .llcrypt unregistered [ 4480.105264] Key type ._llcrypt unregistered [ 4495.494888] alg: No test for adler32 (adler32-zlib) [ 4496.249489] Key type ._llcrypt registered [ 4496.251430] Key type .llcrypt registered [ 4496.330855] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4496.932668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4497.182386] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4497.320945] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4497.323798] LNet: Accept secure, port 988 [ 4498.951721] Key type lgssc registered [ 4499.687571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4504.636671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.016549] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.056204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.240507] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4511.209052] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.246251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.327659] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.548363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4520.377880] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4527.013656] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4531.391460] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4536.179767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4536.299477] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.444158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.412311] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4541.935332] Lustre: lustre-OST0000: deleting orphan objects from 0x0:355 to 0x0:417 [ 4547.137413] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4551.542091] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 4564.268771] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 4573.738633] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 4581.587306] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 4593.120256] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.126305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.130653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.143705] Lustre: Skipped 1 previous similar message [ 4597.785338] Lustre: server umount lustre-OST0000 complete [ 4603.363713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.364961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4603.380183] Lustre: Skipped 1 previous similar message [ 4603.404909] Lustre: Skipped 2 previous similar messages [ 4606.794542] Lustre: server umount lustre-MDT0000 complete [ 4608.481325] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.489561] LustreError: Skipped 1 previous similar message [ 4609.582127] LustreError: 206954:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314796 with bad export cookie 4685167725483128133 [ 4609.584372] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.597194] LustreError: 206954:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4609.853600] Lustre: server umount lustre-MDT0001 complete [ 4618.858065] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4620.937638] Key type lgssc unregistered [ 4621.166180] LNet: 212261:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4621.179291] LNet: Removed LNI 192.168.204.108@tcp [ 4621.936173] Key type .llcrypt unregistered [ 4621.938128] Key type ._llcrypt unregistered [ 4642.226954] alg: No test for adler32 (adler32-zlib) [ 4642.960819] Key type ._llcrypt registered [ 4642.965030] Key type .llcrypt registered [ 4643.068655] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4644.132658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4644.496379] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4644.727643] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4644.732222] LNet: Accept secure, port 988 [ 4646.367123] Key type lgssc registered [ 4647.303833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4653.464211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.729481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.789832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.648384] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4660.196387] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.815248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.119467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.255668] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4673.382970] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4677.635861] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4682.439533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4682.748219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.786017] Lustre: lustre-OST0000: deleting orphan objects from 0x0:355 to 0x0:449 [ 4685.957608] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4691.751370] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4697.352291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.408813] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4697.582477] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4700.386584] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4705.494574] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4706.688434] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4718.222308] Lustre: Permanently deactivating lustre-OST0001 [ 4718.246715] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 4726.821647] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4740.563332] Lustre: Permanently reactivating lustre-OST0001 [ 4740.577748] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 4740.582675] Lustre: Skipped 2 previous similar messages [ 4748.299037] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.305125] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4753.382137] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4753.390279] Lustre: Skipped 2 previous similar messages [ 4753.563442] Lustre: server umount lustre-OST0001 complete [ 4758.497879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.498307] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.503763] Lustre: Skipped 1 previous similar message [ 4758.508853] LustreError: Skipped 1 previous similar message [ 4758.514398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4758.517019] Lustre: Skipped 1 previous similar message [ 4763.489520] Lustre: server umount lustre-OST0000 complete [ 4767.712057] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4767.726386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.736628] Lustre: Skipped 1 previous similar message [ 4767.751457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.510969] Lustre: server umount lustre-MDT0000 complete [ 4773.856122] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.881544] LustreError: Skipped 1 previous similar message [ 4775.171513] LustreError: 213700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314961 with bad export cookie 701474675470888805 [ 4775.173128] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.182080] LustreError: 213700:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4775.507772] Lustre: server umount lustre-MDT0001 complete [ 4783.420808] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 4785.566042] Key type lgssc unregistered [ 4785.819495] LNet: 218861:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4785.829566] LNet: Removed LNI 192.168.204.108@tcp [ 4786.463767] Key type .llcrypt unregistered [ 4786.465600] Key type ._llcrypt unregistered [ 4817.952238] alg: No test for adler32 (adler32-zlib) [ 4818.704613] Key type ._llcrypt registered [ 4818.706557] Key type .llcrypt registered [ 4818.847548] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 4831.260885] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 4832.149662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4832.492190] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 4832.664284] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 4832.667969] LNet: Accept secure, port 988 [ 4834.319158] Key type lgssc registered [ 4835.038178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4841.959864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4847.045626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4851.859979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.727803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4862.501462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4862.554489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.866625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4863.895350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4863.981749] Lustre: lustre-MDT0000: new disk, initializing [ 4864.104933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.139400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4868.907398] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4877.932423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.988746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4878.157638] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4878.224809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4878.237091] Lustre: Skipped 1 previous similar message [ 4878.311912] Lustre: lustre-MDT0001: new disk, initializing [ 4878.400093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.436090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4878.440990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4881.655828] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4889.302896] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4894.156977] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4900.080150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4900.149782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4900.155458] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4900.309987] Lustre: lustre-OST0000: new disk, initializing [ 4900.322526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4900.388045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.019247] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4906.538165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4906.548793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4911.973277] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4916.339090] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4916.540494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4919.772493] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4919.939520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4921.824239] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.828993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.844683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.337492] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.340333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.354324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.357246] Lustre: Skipped 1 previous similar message [ 4927.475350] Lustre: server umount lustre-OST0000 complete [ 4932.577298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.582315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.593479] Lustre: Skipped 1 previous similar message [ 4932.610251] Lustre: Skipped 1 previous similar message [ 4936.396166] Lustre: server umount lustre-MDT0000 complete [ 4937.706498] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.714778] LustreError: Skipped 1 previous similar message [ 4939.114060] LustreError: 226434:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315125 with bad export cookie 304388723867143798 [ 4939.120779] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.318385] Lustre: server umount lustre-MDT0001 complete [ 4945.290545] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:12:10 (1761315130) [ 4951.870727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.310199] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.380578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.426539] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4957.668495] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.786208] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.758670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.055090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4969.468556] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4976.344962] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4981.174417] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4986.706092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.920626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.978687] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 4996.072695] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5008.203940] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5014.358517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5025.337642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5037.963029] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5037.966225] Lustre: Skipped 1 previous similar message [ 5055.948723] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5055.954770] Lustre: Skipped 1 previous similar message [ 5091.507649] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5091.514269] Lustre: Skipped 4 previous similar messages [ 5161.285229] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5161.290469] Lustre: Skipped 8 previous similar messages [ 5176.294219] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5176.295349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.322373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5176.378951] Lustre: Skipped 1 previous similar message [ 5176.384392] Lustre: Skipped 1 previous similar message [ 5181.413569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5181.424324] Lustre: Skipped 1 previous similar message [ 5181.722931] Lustre: server umount lustre-OST0000 complete [ 5185.507057] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5185.515512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.902029] Lustre: server umount lustre-MDT0000 complete [ 5191.650743] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.662978] LustreError: Skipped 1 previous similar message [ 5194.419435] LustreError: 230138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315381 with bad export cookie 304388723867144645 [ 5194.421063] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.434164] LustreError: 230138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5204.974809] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5207.358474] Key type lgssc unregistered [ 5207.606607] LNet: 232418:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5207.615192] LNet: Removed LNI 192.168.204.108@tcp [ 5208.372090] Key type .llcrypt unregistered [ 5208.379378] Key type ._llcrypt unregistered [ 5228.072745] alg: No test for adler32 (adler32-zlib) [ 5228.826162] Key type ._llcrypt registered [ 5228.827819] Key type .llcrypt registered [ 5228.907363] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5229.849673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5230.249677] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 5230.435203] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 5230.442877] LNet: Accept secure, port 988 [ 5232.088205] Key type lgssc registered [ 5232.848561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5240.316112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.753997] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.823940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5246.073436] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 5246.951456] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.068188] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.871631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.419887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.392367] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 5268.689546] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5274.396371] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5280.789560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5281.117829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5285.138941] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_default_debug -1 all 8 [ 5292.288308] Lustre: DEBUG MARKER: oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5310.947547] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.950232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.974463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.972326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.972988] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.973711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.982954] Lustre: Skipped 1 previous similar message [ 5314.073464] Lustre: server umount lustre-OST0000 complete [ 5321.184430] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5321.189981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.213706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.836995] Lustre: server umount lustre-MDT0000 complete [ 5327.193888] LustreError: 236600:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315513 with bad export cookie 6787021602655681713 [ 5327.193888] LustreError: 233864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315513 with bad export cookie 6787021602655681713 [ 5327.194192] LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5327.479514] Lustre: server umount lustre-MDT0001 complete [ 5338.682620] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5341.542892] Key type lgssc unregistered [ 5341.864665] LNet: 237693:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5341.881142] LNet: Removed LNI 192.168.204.108@tcp [ 5342.670546] Key type .llcrypt unregistered [ 5342.672294] Key type ._llcrypt unregistered [ 5365.397684] alg: No test for adler32 (adler32-zlib) [ 5366.154404] Key type ._llcrypt registered [ 5366.155865] Key type .llcrypt registered [ 5366.313673] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5366.624962] Key type .llcrypt unregistered [ 5366.637347] Key type ._llcrypt unregistered [ 5389.638182] alg: No test for adler32 (adler32-zlib) [ 5390.400463] Key type ._llcrypt registered [ 5390.402426] Key type .llcrypt registered [ 5390.529285] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing set_hostid [ 5404.586513] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing load_modules_local [ 5406.272505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5406.847665] Lustre: Lustre: Build Version: 2.15.7_9_g374ce7d [ 5407.069975] LNet: Added LNI 192.168.204.108@tcp [8/256/0/180] [ 5407.073487] LNet: Accept secure, port 988 [ 5408.735731] Key type lgssc registered [ 5410.206927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5420.488802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5427.372635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5432.494378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5437.522960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5438.956210] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5315 sec ========== 10:20:24 (1761315624) [ 5472.405960] Lustre: DEBUG MARKER: oleg408-server.virtnet: executing unload_modules_local [ 5474.723762] Key type lgssc unregistered [ 5474.955046] LNet: 243950:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5474.959809] LNet: Removed LNI 192.168.204.108@tcp [ 5475.615510] Key type .llcrypt unregistered [ 5475.617667] Key type ._llcrypt unregistered