[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 530404047 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003272] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007790] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.011101] LSM: Security Framework initializing [ 0.012067] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.015082] *** VALIDATE selinux *** [ 0.024388] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028786] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031164] *** VALIDATE tmpfs *** [ 0.033379] *** VALIDATE proc *** [ 0.034305] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.037215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039055] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.046057] debug: unmapping init [mem 0xffffffff92859000-0xffffffff92860fff] [ 0.048201] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050020] ... version: 2 [ 0.051011] ... bit width: 48 [ 0.052012] ... generic registers: 4 [ 0.052977] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055009] ... event mask: 000000070000000f [ 0.056329] rcu: Hierarchical SRCU implementation. [ 0.058591] smp: Bringing up secondary CPUs ... [ 0.059571] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.063011] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.064895] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147022] node 0 deferred pages initialised in 81ms [ 0.151010] devtmpfs: initialized [ 0.152267] x86/mm: Memory block size: 128MB [ 0.154743] gcov: version magic: 0x41383552 [ 0.158262] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163358] pinctrl core: initialized pinctrl subsystem [ 0.165170] [ 0.165715] ************************************************************* [ 0.167012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ** ** [ 0.172015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176015] ** This means that this kernel is built to expose internal ** [ 0.178014] ** IOMMU data structures, which may compromise security on ** [ 0.180012] ** your system. ** [ 0.182012] ** ** [ 0.184015] ** If you see this message and you are not debugging the ** [ 0.186015] ** kernel, report this immediately to your vendor! ** [ 0.188019] ** ** [ 0.190012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192015] ************************************************************* [ 0.195262] NET: Registered protocol family 16 [ 0.197526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206125] cpuidle: using governor menu [ 0.207520] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209629] PCI: Using configuration type 1 for base access [ 0.212166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228066] cryptd: max_cpu_qlen set to 1000 [ 0.230256] ACPI: Added _OSI(Module Device) [ 0.232021] ACPI: Added _OSI(Processor Device) [ 0.234018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240981] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249386] ACPI: Interpreter enabled [ 0.250058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251011] ACPI: Using IOAPIC for interrupt routing [ 0.253103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263773] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277378] acpiphp: Slot [2] registered [ 0.279136] acpiphp: Slot [5] registered [ 0.281147] acpiphp: Slot [6] registered [ 0.283135] acpiphp: Slot [7] registered [ 0.284158] acpiphp: Slot [8] registered [ 0.286149] acpiphp: Slot [9] registered [ 0.288140] acpiphp: Slot [10] registered [ 0.289156] acpiphp: Slot [3] registered [ 0.291128] acpiphp: Slot [4] registered [ 0.293119] acpiphp: Slot [11] registered [ 0.294099] acpiphp: Slot [12] registered [ 0.296169] acpiphp: Slot [13] registered [ 0.297110] acpiphp: Slot [14] registered [ 0.299154] acpiphp: Slot [15] registered [ 0.300134] acpiphp: Slot [16] registered [ 0.302118] acpiphp: Slot [17] registered [ 0.304130] acpiphp: Slot [18] registered [ 0.305109] acpiphp: Slot [19] registered [ 0.307117] acpiphp: Slot [20] registered [ 0.308139] acpiphp: Slot [21] registered [ 0.310119] acpiphp: Slot [22] registered [ 0.311120] acpiphp: Slot [23] registered [ 0.313113] acpiphp: Slot [24] registered [ 0.315117] acpiphp: Slot [25] registered [ 0.316118] acpiphp: Slot [26] registered [ 0.317076] acpiphp: Slot [27] registered [ 0.318096] acpiphp: Slot [28] registered [ 0.319090] acpiphp: Slot [29] registered [ 0.321109] acpiphp: Slot [30] registered [ 0.323123] acpiphp: Slot [31] registered [ 0.324088] PCI host bridge to bus 0000:00 [ 0.326022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.338027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343111] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346263] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355979] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360125] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373707] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379938] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405069] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456219] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.475021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515323] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.557683] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.566019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.598020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.609716] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.652879] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.659015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.682026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.693470] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.696521] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.701455] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.703406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706326] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.710124] iommu: Default domain type: Passthrough [ 0.712476] SCSI subsystem initialized [ 0.714185] ACPI: bus type USB registered [ 0.716113] usbcore: registered new interface driver usbfs [ 0.718092] usbcore: registered new interface driver hub [ 0.720133] usbcore: registered new device driver usb [ 0.722185] pps_core: LinuxPPS API ver. 1 registered [ 0.723013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727071] PTP clock support registered [ 0.729052] EDAC MC: Ver: 3.0.0 [ 0.731177] PCI: Using ACPI for IRQ routing [ 0.733568] NetLabel: Initializing [ 0.735017] NetLabel: domain hash size = 128 [ 0.737013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.740092] NetLabel: unlabeled traffic allowed by default [ 0.742178] vgaarb: loaded [ 0.744290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.749300] clocksource: Switched to clocksource kvm-clock [ 0.846597] VFS: Disk quotas dquot_6.6.0 [ 0.847697] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.849306] *** VALIDATE ramfs *** [ 0.850111] *** VALIDATE hugetlbfs *** [ 0.851428] pnp: PnP ACPI init [ 0.853943] pnp: PnP ACPI: found 6 devices [ 0.873431] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.876753] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.878590] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.880903] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883643] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.886149] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.889157] NET: Registered protocol family 2 [ 0.891729] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.896953] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.900771] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.906313] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910155] TCP: Hash tables configured (established 65536 bind 65536) [ 0.913091] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.916251] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.919182] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.922505] NET: Registered protocol family 1 [ 0.926156] RPC: Registered named UNIX socket transport module. [ 0.928683] RPC: Registered udp transport module. [ 0.930569] RPC: Registered tcp transport module. [ 0.932407] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935284] NET: Registered protocol family 44 [ 0.937050] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.939266] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.941553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.944174] PCI: CLS 0 bytes, default 64 [ 0.946170] Unpacking initramfs... [ 2.375933] debug: unmapping init [mem 0xffff9478bcc54000-0xffff9478bffbffff] [ 2.379620] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.382049] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.384305] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.871043] Initialise system trusted keyrings [ 2.872181] Key type blacklist registered [ 2.873606] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.881906] zbud: loaded [ 2.884876] *** VALIDATE nfs *** [ 2.885898] *** VALIDATE nfs4 *** [ 2.886988] pstore: using deflate compression [ 2.890390] Platform Keyring initialized [ 2.973789] NET: Registered protocol family 38 [ 2.975047] Key type asymmetric registered [ 2.976291] Asymmetric key parser 'x509' registered [ 2.978037] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.980112] io scheduler mq-deadline registered [ 2.981360] io scheduler kyber registered [ 2.982399] io scheduler bfq registered [ 2.983697] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.986423] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.989409] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.991760] ACPI: Power Button [PWRF] [ 2.995964] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.001375] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.012586] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.025363] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.042938] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070347] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.097996] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.105368] Non-volatile memory driver v1.3 [ 3.106981] Linux agpgart interface v0.103 [ 3.142925] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.145726] vda: detected capacity change from 0 to 34836480 [ 3.161464] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.163981] vdb: detected capacity change from 0 to 1073741824 [ 3.181811] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.184501] vdc: detected capacity change from 0 to 2621440000 [ 3.195853] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.197791] vdd: detected capacity change from 0 to 2621440000 [ 3.213507] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.215129] vde: detected capacity change from 0 to 4294967296 [ 3.228046] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.230159] vdf: detected capacity change from 0 to 4294967296 [ 3.235099] libphy: Fixed MDIO Bus: probed [ 3.241742] usbcore: registered new interface driver usbserial_generic [ 3.244139] usbserial: USB Serial support registered for generic [ 3.246940] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.251550] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.253632] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.256444] mousedev: PS/2 mouse device common for all mice [ 3.259776] rtc_cmos 00:05: RTC can wake from S4 [ 3.262413] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.265512] rtc_cmos 00:05: registered as rtc0 [ 3.267786] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.268821] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.272489] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.275481] intel_pstate: CPU model not supported [ 3.281145] hid: raw HID events driver (C) Jiri Kosina [ 3.284294] usbcore: registered new interface driver usbhid [ 3.287582] usbhid: USB HID core driver [ 3.289291] drop_monitor: Initializing network drop monitor service [ 3.292379] Initializing XFRM netlink socket [ 3.295450] NET: Registered protocol family 10 [ 3.299664] Segment Routing with IPv6 [ 3.301741] NET: Registered protocol family 17 [ 3.304252] mpls_gso: MPLS GSO support [ 3.310422] RAS: Correctable Errors collector initialized. [ 3.312397] AVX version of gcm_enc/dec engaged. [ 3.314170] AES CTR mode by8 optimization enabled [ 3.383519] sched_clock: Marking stable (3383495680, 0)->(4306749017, -923253337) [ 3.386877] registered taskstats version 1 [ 3.388727] Loading compiled-in X.509 certificates [ 3.390117] zswap: loaded using pool lzo/zbud [ 3.412769] Key type big_key registered [ 3.422986] Key type encrypted registered [ 3.424538] ima: No TPM chip found, activating TPM-bypass! [ 3.426264] ima: Allocated hash algorithm: sha1 [ 3.427696] ima: No architecture policies found [ 3.429150] evm: Initialising EVM extended attributes: [ 3.430784] evm: security.selinux [ 3.431771] evm: security.ima [ 3.432660] evm: security.capability [ 3.433898] evm: HMAC attrs: 0x1 [ 3.435640] rtc_cmos 00:05: setting system clock to 2026-04-14 18:30:16 UTC (1776191416) [ 3.440609] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.444902] debug: unmapping init [mem 0xffffffff92582000-0xffffffff92858fff] [ 3.459135] Write protecting the kernel read-only data: 28672k [ 3.462483] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 3.465406] debug: unmapping init [mem 0xffffffff91514000-0xffffffff915fffff] [ 3.497911] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.507545] systemd[1]: Detected virtualization kvm. [ 3.509418] systemd[1]: Detected architecture x86-64. [ 3.511278] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.538633] systemd[1]: No hostname configured. [ 3.542616] systemd[1]: Set hostname to . [ 3.545569] random: systemd: uninitialized urandom read (16 bytes read) [ 3.549081] systemd[1]: Initializing machine ID from random generator. [ 3.689701] random: systemd: uninitialized urandom read (16 bytes read) [ 3.692050] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.696480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699055] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.702633] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.303222] device-mapper: uevent: version 1.0.3 [ 4.305412] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.939445] virtio_net virtio0 ens2: renamed from eth0 [ 4.959973] scsi host0: ata_piix [ 4.965193] scsi host1: ata_piix [ 4.967195] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.969629] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.990855] random: fast init done [ 9.189467] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.938079] random: crng init done [ 9.939544] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.272814] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.502818] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.756902] SELinux: Disabled at runtime. [ 11.821620] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.830178] systemd[1]: Detected virtualization kvm. [ 11.832268] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.344564] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.347758] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.352136] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.355402] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.359377] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.367805] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.371767] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.566076] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.887594] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.291749] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.306984] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.494511] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.504232] EDAC sbridge: Ver: 1.1.2 [ 15.333289] Key type dns_resolver registered [ 15.667988] NFS: Registering the id_resolver key type [ 15.669676] Key type id_resolver registered [ 15.670827] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg215-server login: [ 25.770137] hrtimer: interrupt took 2453346 ns [ 61.420185] libcfs: loading out-of-tree module taints kernel. [ 61.457446] alg: No test for adler32 (adler32-zlib) [ 62.215770] Key type ._llcrypt registered [ 62.218378] Key type .llcrypt registered [ 62.425375] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 86.207929] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 87.994284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.768758] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 89.521535] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 89.524846] LNet: Accept secure, port 988 [ 91.311687] Key type lgssc registered [ 93.138633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.428247] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.881729] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.109127] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.425823] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.558858] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.169164] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 179.126443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.287264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.759692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 180.809861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 180.951873] Lustre: lustre-MDT0000: new disk, initializing [ 181.036879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.064338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.834634] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 201.066560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.152227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.207868] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 201.223914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 201.227234] Lustre: Skipped 1 previous similar message [ 201.293250] Lustre: lustre-MDT0001: new disk, initializing [ 201.344303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.372265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.388912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.063762] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 222.307987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.470729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.481606] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 222.912106] Lustre: lustre-OST0000: new disk, initializing [ 222.923851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 223.055287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.009685] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 230.967296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.988680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.635274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.748956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.758426] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 244.901530] Lustre: lustre-OST0001: new disk, initializing [ 244.919370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 245.056433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.597225] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 251.418257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.468068] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.755394] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.628537] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 279.764154] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.000518] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing yml_node [ 289.533550] Lustre: DEBUG MARKER: Client: 2.15.8.10 [ 292.089236] Lustre: DEBUG MARKER: MDS: 2.15.8.10 [ 294.729132] Lustre: DEBUG MARKER: OSS: 2.15.8.10 [ 296.335303] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:35:08 EDT 2026 [ 305.026854] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 306.947607] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 324.578298] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.585891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.611099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.159709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.160657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.170984] Lustre: Skipped 1 previous similar message [ 328.194212] Lustre: Skipped 3 previous similar messages [ 333.283072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.289682] Lustre: Skipped 3 previous similar messages [ 338.400099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.400942] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 338.406648] Lustre: Skipped 3 previous similar messages [ 338.592281] Lustre: server umount lustre-MDT0000 complete [ 343.528096] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.549742] LustreError: Skipped 3 previous similar messages [ 343.562619] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191756 with bad export cookie 17979279412828284307 [ 343.571761] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.577060] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.641620] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.641778] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.669264] LustreError: Skipped 1 previous similar message [ 348.693609] Lustre: Skipped 1 previous similar message [ 348.701098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.706335] Lustre: Skipped 1 previous similar message [ 349.728892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.743734] LustreError: Skipped 1 previous similar message [ 349.947344] Lustre: server umount lustre-MDT0001 complete [ 353.782437] Lustre: server umount lustre-OST0000 complete [ 357.850961] Lustre: server umount lustre-OST0001 complete [ 364.696382] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 372.844618] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 384.344665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.713526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.351772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.887620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.959821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.037778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.290661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 410.331390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 410.401359] Lustre: lustre-MDT0000: new disk, initializing [ 410.463828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.479280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.018554] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 425.997149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 426.116560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.275399] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 426.338877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 426.344201] Lustre: Skipped 1 previous similar message [ 426.494459] Lustre: lustre-MDT0001: new disk, initializing [ 426.613051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.659289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.673289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.190732] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 440.463522] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 446.381594] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 453.493744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 453.570602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.576943] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 454.018699] Lustre: lustre-OST0000: new disk, initializing [ 454.033946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 454.198973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.053408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.074057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.640970] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 471.547985] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.436813] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 477.580027] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.087468] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 482.337537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 486.880636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.884600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.921731] Lustre: Skipped 2 previous similar messages [ 486.951987] Lustre: Skipped 3 previous similar messages [ 490.744532] Lustre: server umount lustre-OST0000 complete [ 497.128132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.002620] Lustre: server umount lustre-MDT0000 complete [ 502.241858] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.273693] LustreError: Skipped 1 previous similar message [ 506.622461] LustreError: 14345:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776191919 with bad export cookie 17979279412828285763 [ 506.624452] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.628907] LustreError: 14345:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.396925] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 14:38:44 (1776191924) [ 521.025220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.537693] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.572592] LustreError: Skipped 2 previous similar messages [ 521.684761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.561523] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 531.938649] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.975926] LustreError: Skipped 1 previous similar message [ 536.232440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.229486] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 551.011419] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 557.711981] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 564.592169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.431893] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 578.444446] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 582.096295] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 591.843521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.848377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.863763] Lustre: Skipped 2 previous similar messages [ 591.880414] Lustre: Skipped 4 previous similar messages [ 595.354516] Lustre: server umount lustre-MDT0000 complete [ 595.364723] Lustre: Skipped 1 previous similar message [ 596.971168] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.996548] LustreError: Skipped 3 previous similar messages [ 599.187889] LustreError: 18740:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192012 with bad export cookie 17979279412828286617 [ 599.201334] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.247243] Lustre: 21865:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192016/real 1776192016] req@00000000d17a0051 x1862471780491264/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192022 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 630.967699] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 641.416442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.967914] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.083679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.099103] Lustre: Skipped 2 previous similar messages [ 647.508124] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 656.372440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.610952] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 669.530430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.755740] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 683.003641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 683.097368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.108793] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 683.313567] Lustre: lustre-OST0001: new disk, initializing [ 683.318843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 684.489038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 684.510251] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 688.186639] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 698.009846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 701.810922] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 718.305587] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.313839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.330130] Lustre: Skipped 1 previous similar message [ 718.344347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.540459] Lustre: server umount lustre-MDT0000 complete [ 721.543035] Lustre: Skipped 2 previous similar messages [ 725.127785] LustreError: 23232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192138 with bad export cookie 17979279412828287464 [ 725.128936] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.139050] LustreError: 23232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 732.575165] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192138/real 1776192138] req@0000000010f45e02 x1862471780522816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192145 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 735.712650] Lustre: 27065:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192142/real 1776192142] req@00000000962f4143 x1862471780522944/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192148 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 737.761579] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192143/real 1776192143] req@00000000ee6a1828 x1862471780523008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192150 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 755.956785] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 765.980155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.343465] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.354984] LustreError: Skipped 3 previous similar messages [ 771.436333] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 781.581201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.158636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 782.165433] Lustre: Skipped 4 previous similar messages [ 786.716955] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 795.441702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 800.565070] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 809.499566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 813.206986] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 824.042263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 830.238359] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 830.243386] Lustre: Skipped 1 previous similar message [ 844.263186] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.291089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.300689] Lustre: Skipped 4 previous similar messages [ 844.308585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.311790] Lustre: Skipped 5 previous similar messages [ 850.528562] Lustre: server umount lustre-MDT0000 complete [ 850.531385] Lustre: Skipped 3 previous similar messages [ 855.619652] LustreError: 28478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192268 with bad export cookie 17979279412828288710 [ 855.632873] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.135565] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192269/real 1776192269] req@0000000055c3456d x1862471780551104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192276 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 863.187237] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 882.820640] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 14:44:54 (1776192294) [ 914.270772] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 926.436645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.797698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.806324] LustreError: Skipped 13 previous similar messages [ 932.743599] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 941.308753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.690385] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 958.517576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 966.023888] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 976.594606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.214371] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 989.746570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 992.947783] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1002.977325] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1002.980555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.986795] Lustre: Skipped 4 previous similar messages [ 1002.989702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.993720] Lustre: Skipped 4 previous similar messages [ 1009.213058] Lustre: server umount lustre-MDT0000 complete [ 1009.217775] Lustre: Skipped 3 previous similar messages [ 1012.737228] LustreError: 34325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192425 with bad export cookie 17979279412828289781 [ 1012.749875] LustreError: 34325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1012.752820] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.831157] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192426/real 1776192426] req@00000000dc877781 x1862471780577152/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192433 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1020.853599] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1034.245479] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 14:47:26 (1776192446) [ 1047.521697] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1057.452353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.027854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.040416] Lustre: Skipped 6 previous similar messages [ 1062.580761] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1070.669767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.790650] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1084.145588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.154205] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1097.211711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.543648] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1109.647327] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1132.012867] LustreError: 39520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192545 with bad export cookie 17979279412828290852 [ 1132.022193] LustreError: 39520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1132.023207] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.540091] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.548245] Lustre: Skipped 9 previous similar messages [ 1146.335251] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1156.934706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.305483] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0xf9833b5e1f1fbf24 to 0xf9833b5e1f1fc38b [ 1165.323478] Lustre: MGC192.168.202.115@tcp: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 1166.100284] BUG: spinlock bad magic on CPU#3, lctl/43454 [ 1166.106924] lock: 0xffff9478065f4d00, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1166.112380] CPU: 3 PID: 43454 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1166.130399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 1166.139643] Call Trace: [ 1166.140272] ? dump_stack+0xbb/0x10e [ 1166.141100] ? spin_dump.cold.3+0x32/0x65 [ 1166.154740] ? do_raw_spin_lock+0xce/0x190 [ 1166.160255] ? _raw_spin_lock+0x21/0x30 [ 1166.172776] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 1166.180359] ? __seccomp_filter+0x4b/0x680 [ 1166.197267] ? full_proxy_write+0x5e/0xa0 [ 1166.200405] ? __vfs_write+0x1c/0x60 [ 1166.209071] ? vfs_write+0xd8/0x2b0 [ 1166.215101] ? ksys_write+0x66/0x120 [ 1166.216256] ? __x64_sys_write+0x1e/0x30 [ 1166.227111] ? do_syscall_64+0xc1/0x440 [ 1166.230255] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1170.615400] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1174.513223] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1179.404793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.133264] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1184.753687] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1184.761933] Lustre: Skipped 1 previous similar message [ 1191.278981] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1192.655146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1198.118185] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1199.674470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1205.226524] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1214.350819] LustreError: 39518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192627 with bad export cookie 17979279412828291979 [ 1214.351531] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.358524] LustreError: 39518:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1222.559744] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192628/real 1776192628] req@000000000ad5caef x1862471780621632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192635 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1222.611730] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1240.276512] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 14:50:51 (1776192651) [ 1258.195699] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1269.562032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.903865] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.914606] LustreError: Skipped 33 previous similar messages [ 1275.613968] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1285.916432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.929249] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1298.798850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.901928] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1314.534750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.882310] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1329.218875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1333.541569] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1333.551675] Lustre: Skipped 2 previous similar messages [ 1355.752060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.758054] Lustre: Skipped 39 previous similar messages [ 1360.409429] Lustre: server umount lustre-MDT0000 complete [ 1360.412796] Lustre: Skipped 9 previous similar messages [ 1363.445331] LustreError: 47281:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776192776 with bad export cookie 17979279412828292784 [ 1363.455323] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1373.151787] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192779/real 1776192779] req@00000000e356e05c x1862471780652800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776192786 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1373.168459] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1391.265172] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 14:53:23 (1776192803) [ 1394.731495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1397.222173] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 14:53:28 (1776192808) [ 1408.912688] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 1419.926339] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1428.592332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1433.854427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.272276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.388909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.766335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1441.932208] Lustre: lustre-MDT0000: new disk, initializing [ 1442.159665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1447.519618] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1458.680398] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1465.439657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1465.536355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.539913] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1465.784265] Lustre: lustre-OST0000: new disk, initializing [ 1465.788087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1465.800939] Lustre: Skipped 1 previous similar message [ 1470.159895] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1479.782717] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1482.071500] Lustre: DEBUG MARKER: create test files [ 1513.444086] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.451495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.474374] Lustre: Skipped 10 previous similar messages [ 1526.239312] Lustre: 55612:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776192933/real 1776192933] req@000000007f43e81b x1862471780662336/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776192939 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1526.266447] Lustre: 55612:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 1538.647465] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1542.083745] Key type lgssc unregistered [ 1542.381395] LNet: 56217:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.389514] LNet: Removed LNI 192.168.202.115@tcp [ 1543.344262] Key type .llcrypt unregistered [ 1543.347795] Key type ._llcrypt unregistered [ 1574.081666] alg: No test for adler32 (adler32-zlib) [ 1574.842079] Key type ._llcrypt registered [ 1574.845316] Key type .llcrypt registered [ 1574.940461] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1576.817141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.333876] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1577.589111] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1577.601511] LNet: Accept secure, port 988 [ 1579.295151] Key type lgssc registered [ 1580.813318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.830921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.848903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.868748] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1603.943939] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.170195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.895108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.005786] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1621.107111] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1621.242444] Lustre: 59089:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1621.248816] Lustre: 59089:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1621.262429] Lustre: 59089:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1621.264992] Lustre: 59089:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1621.273317] Lustre: 59089:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1621.276744] Lustre: 59089:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1625.612902] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1631.069884] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1635.775882] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1647.073480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.073666] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.094075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.449117] Lustre: server umount lustre-OST0000 complete [ 1659.359234] Lustre: 60106:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193066/real 1776193066] req@000000000247354d x1862473340689216/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193072 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1659.510720] Lustre: server umount lustre-MDT0000 complete [ 1671.605202] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1674.336878] Key type lgssc unregistered [ 1674.588180] LNet: 60710:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.597044] LNet: Removed LNI 192.168.202.115@tcp [ 1675.550573] Key type .llcrypt unregistered [ 1675.553622] Key type ._llcrypt unregistered [ 1705.400393] alg: No test for adler32 (adler32-zlib) [ 1706.156649] Key type ._llcrypt registered [ 1706.163542] Key type .llcrypt registered [ 1706.372289] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1707.776487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.514765] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1708.747089] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1708.749406] LNet: Accept secure, port 988 [ 1710.480085] Key type lgssc registered [ 1711.596945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.083816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.721452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.126067] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1735.879426] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1743.138938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.403302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.626298] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1752.637980] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1752.731451] Lustre: 63652:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1752.741777] Lustre: 63652:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1752.750027] Lustre: 63652:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1752.758632] Lustre: 63652:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1752.766232] Lustre: 63652:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1752.775695] Lustre: 63652:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1757.004734] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1762.217659] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1773.026589] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.038950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.062849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.203523] Lustre: server umount lustre-OST0000 complete [ 1787.362651] Lustre: 64617:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776193194/real 1776193194] req@00000000e99edb5d x1862473478052224/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776193200 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1787.588497] Lustre: server umount lustre-MDT0000 complete [ 1800.255205] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 1803.160046] Key type lgssc unregistered [ 1803.518928] LNet: 65221:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.526772] LNet: Removed LNI 192.168.202.115@tcp [ 1804.538769] Key type .llcrypt unregistered [ 1804.543887] Key type ._llcrypt unregistered [ 1823.871717] alg: No test for adler32 (adler32-zlib) [ 1824.625268] Key type ._llcrypt registered [ 1824.627707] Key type .llcrypt registered [ 1824.757241] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 1840.582506] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 1842.251980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1843.132718] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1843.364560] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 1843.368023] LNet: Accept secure, port 988 [ 1845.039963] Key type lgssc registered [ 1846.423433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.666035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1866.682499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1873.779665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1880.698033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1887.612382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1887.716925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.048617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1889.104460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1889.242581] Lustre: lustre-MDT0000: new disk, initializing [ 1889.311323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.327249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1893.856703] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1905.211941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1905.288052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.460253] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1905.566671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1905.587655] Lustre: Skipped 1 previous similar message [ 1905.729714] Lustre: lustre-MDT0001: new disk, initializing [ 1905.877392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.946276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1905.962645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1911.594832] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1923.110804] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1930.144853] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.206268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1938.291904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.298202] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1938.549443] Lustre: lustre-OST0000: new disk, initializing [ 1938.553888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1938.670254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.730743] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 1945.142707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1945.150973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1955.835937] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1962.681838] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1963.118897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1968.651211] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1968.927787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1971.168837] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.174138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.184363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.188795] Lustre: Skipped 1 previous similar message [ 1975.782051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.795015] Lustre: Skipped 1 previous similar message [ 1977.279077] Lustre: server umount lustre-OST0000 complete [ 1982.945593] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.949036] LustreError: Skipped 1 previous similar message [ 1982.971891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.992870] Lustre: Skipped 1 previous similar message [ 1983.000691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.028318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.034342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.567506] Lustre: server umount lustre-MDT0000 complete [ 1991.138931] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.153235] LustreError: Skipped 1 previous similar message [ 1993.442914] LustreError: 68530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193406 with bad export cookie 10380843701293769864 [ 1993.450016] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1993.455340] LustreError: 68530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1993.723311] Lustre: server umount lustre-MDT0001 complete [ 2003.745394] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 15:03:35 (1776193415) [ 2015.244974] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2029.949797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2057.324748] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 2068.800590] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2079.812322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2087.720347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2094.040386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2101.665530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.510412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.579299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.786724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2109.825803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2109.927728] Lustre: lustre-MDT0000: new disk, initializing [ 2110.048510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.069941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2114.123953] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2124.690314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2124.825549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.039945] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2125.151472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2125.168167] Lustre: Skipped 1 previous similar message [ 2125.260540] Lustre: lustre-MDT0001: new disk, initializing [ 2125.439713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.534197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2125.579836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2130.585246] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2142.165317] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2148.770544] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2155.059302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2155.128735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.136790] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2155.350368] Lustre: lustre-OST0000: new disk, initializing [ 2155.356078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2155.438486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.619058] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2163.306198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2163.329236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2170.492633] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.430484] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2176.733951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2182.462914] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2182.719791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2189.280327] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2189.280758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.289792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.289799] Lustre: Skipped 1 previous similar message [ 2189.319327] Lustre: Skipped 1 previous similar message [ 2191.945840] Lustre: server umount lustre-OST0000 complete [ 2199.549425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.584884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.600524] Lustre: Skipped 2 previous similar messages [ 2203.485729] Lustre: server umount lustre-MDT0000 complete [ 2204.649373] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.672439] LustreError: Skipped 1 previous similar message [ 2205.665561] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.687465] LustreError: Skipped 1 previous similar message [ 2207.759546] LustreError: 76832:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193620 with bad export cookie 10380843701293770711 [ 2207.774399] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.696236] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 15:07:08 (1776193628) [ 2223.344539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.931586] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.061084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.759313] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2229.218066] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.226815] LustreError: Skipped 1 previous similar message [ 2238.301773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.014990] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2254.920543] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2261.590799] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.726471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.081354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2270.087910] Lustre: Skipped 1 previous similar message [ 2275.656177] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2284.639749] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2288.909691] Lustre: 84132:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 693638b5-7675-49f0-b1ff-793338e7afd4 at adminstrative request [ 2293.739951] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.768065] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2299.873745] Lustre: MGS: Client 693638b5-7675-49f0-b1ff-793338e7afd4 (at 0@lo) reconnecting [ 2299.898781] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x90102a58a4776b34 to 0x90102a58a4776dd4 [ 2299.915403] Lustre: MGC192.168.202.115@tcp: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 2326.782700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2326.896723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.900228] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2327.040349] Lustre: lustre-OST0001: new disk, initializing [ 2327.045163] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2332.739728] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2336.929707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2336.958079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2344.219517] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2347.491508] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.500070] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.501910] LustreError: Skipped 1 previous similar message [ 2347.503246] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2347.522693] Lustre: Skipped 2 previous similar messages [ 2353.530078] Lustre: server umount lustre-OST0001 complete [ 2353.538154] Lustre: Skipped 1 previous similar message [ 2357.735056] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.755818] LustreError: Skipped 2 previous similar messages [ 2362.339298] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.975235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.994075] Lustre: Skipped 3 previous similar messages [ 2374.396262] Lustre: server umount lustre-MDT0000 complete [ 2374.406363] Lustre: Skipped 1 previous similar message [ 2378.213316] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.239679] LustreError: Skipped 3 previous similar messages [ 2379.515740] LustreError: 81371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193792 with bad export cookie 10380843701293772244 [ 2379.536735] LustreError: 81371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2379.541831] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.588927] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 15:10:00 (1776193800) [ 2390.063792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2391.652806] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 15:10:03 (1776193803) [ 2393.258960] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2394.856779] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 15:10:07 (1776193807) [ 2396.513864] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2398.460967] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 15:10:10 (1776193810) [ 2400.153504] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2405.452422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.883442] loop: module loaded [ 2406.898315] loop0: detected capacity change from 0 to 4294967296 [ 2406.974226] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2408.724473] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2412.703377] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2416.856324] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2425.695278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2430.021542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.374199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2438.902652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.360921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.403171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.406238] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2446.413696] LustreError: 88581:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2446.419760] LustreError: 88581:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2446.423603] LustreError: 88581:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2446.428256] LustreError: 88581:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2465.936588] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 2474.342716] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 2484.993243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.568372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.158031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2505.668964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2513.503370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2513.583368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.823301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2513.866715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2514.040617] Lustre: lustre-MDT0000: new disk, initializing [ 2514.178494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.187874] Lustre: Skipped 1 previous similar message [ 2514.212949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2519.406582] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2530.114845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2530.202443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.321385] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2530.660330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2536.269875] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2547.455264] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2554.438481] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.422878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2562.531000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.539983] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2568.649205] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2569.296090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2579.947950] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2585.753704] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2586.107935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2591.531665] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2591.727818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2595.298540] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.300727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.307563] LustreError: Skipped 1 previous similar message [ 2595.308210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.308214] Lustre: Skipped 10 previous similar messages [ 2595.350696] Lustre: Skipped 1 previous similar message [ 2599.940313] Lustre: server umount lustre-OST0000 complete [ 2599.944720] Lustre: Skipped 1 previous similar message [ 2618.335237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2621.920783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.940444] LustreError: Skipped 1 previous similar message [ 2622.909872] LustreError: 91438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194035 with bad export cookie 10380843701293772825 [ 2622.928106] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.944117] LustreError: 91438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2634.441822] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 15:14:06 (1776194046) [ 2641.897813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.480789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2642.496238] Lustre: Skipped 2 previous similar messages [ 2649.394240] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2659.989217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.638851] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2675.144724] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2682.420811] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2690.593503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.517686] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2706.094276] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2714.133303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.228840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2714.238833] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2714.376327] Lustre: lustre-OST0001: new disk, initializing [ 2714.381407] Lustre: Skipped 2 previous similar messages [ 2714.390786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2714.401699] Lustre: Skipped 3 previous similar messages [ 2720.270767] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2722.411206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2722.418992] Lustre: Skipped 2 previous similar messages [ 2722.439974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2731.373829] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2750.556715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2770.533801] Lustre: Failing over lustre-MDT0000 [ 2772.961418] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.974962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.998040] Lustre: Skipped 2 previous similar messages [ 2773.011128] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.022347] LustreError: Skipped 4 previous similar messages [ 2773.072137] Lustre: server umount lustre-MDT0000 complete [ 2773.075766] Lustre: Skipped 2 previous similar messages [ 2781.151479] Lustre: 66742:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776194187/real 1776194187] req@000000008f35d863 x1862473619739904/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776194194 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2781.185130] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.476556] LDISKFS-fs (dm-0): recovery complete [ 2801.479711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.018860] Lustre: Evicted from MGS (at 192.168.202.115@tcp) after server handle changed from 0x90102a58a477736f to 0x90102a58a4780152 [ 2815.076738] Lustre: MGC192.168.202.115@tcp: Connection restored to (at 0@lo) [ 2815.545931] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2819.392132] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2819.393910] Lustre: 101163:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2820.603227] Lustre: 96000:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2820.618513] Lustre: 96000:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 1 previous similar message [ 2820.627348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2820.745788] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2821.068723] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 2821.078516] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2821.421248] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2821.434119] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2821.759163] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2821.765516] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2821.775419] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2821.789768] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 7 previous similar messages [ 2822.473881] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2822.492207] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2822.845015] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2822.859480] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2823.880815] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2823.887871] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 5 previous similar messages [ 2824.528437] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2824.540310] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2824.863299] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2824.877808] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2828.045096] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2828.065276] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 12 previous similar messages [ 2828.708485] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2828.714578] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2829.031153] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2829.036598] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2836.072081] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2836.084188] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 24 previous similar messages [ 2836.726140] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2836.754595] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2837.079155] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2837.093190] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2852.104433] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2852.120455] Lustre: 101197:0:(ldlm_lib.c:1972:extend_recovery_timer()) Skipped 49 previous similar messages [ 2852.768080] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2852.785448] LustreError: 101197:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 49 previous similar messages [ 2853.111134] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2853.122260] LustreError: 101197:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 49 previous similar messages [ 2879.287232] Lustre: 101197:0:(ldlm_lib.c:2280:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2879.308811] Lustre: 101197:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2879.326257] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2879.335159] LustreError: 101197:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2879.357584] Lustre: 101197:0:(ldlm_lib.c:1804:abort_req_replay_queue()) @@@ aborted: req@00000000ebb4127c x1862473603904704/t0(8589934779) o101->7fc1c5a5-5c9e-4525-9774-35ef73fbb9c9@192.168.202.15@tcp:173/0 lens 592/0 e 0 to 0 dl 1776194298 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2879.400227] Lustre: lustre-MDT0000: Denying connection for new client 7fc1c5a5-5c9e-4525-9774-35ef73fbb9c9 (at 192.168.202.15@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2879.412397] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2879.475409] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2879.520159] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2879.533085] Lustre: Skipped 2 previous similar messages [ 2879.541196] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2879.603761] Lustre: lustre-OST0000: deleting orphan objects from 0x0:80 to 0x0:577 [ 2879.604227] Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:545 [ 2897.399714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.409072] Lustre: Skipped 10 previous similar messages [ 2903.382754] Lustre: server umount lustre-OST0000 complete [ 2907.627690] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.628411] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.642537] LustreError: Skipped 73 previous similar messages [ 2907.663244] Lustre: Skipped 5 previous similar messages [ 2912.225508] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2928.862194] LustreError: 95981:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194341 with bad export cookie 10380843701293810002 [ 2928.870692] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.892058] LustreError: 95981:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2938.370980] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 15:19:10 (1776194350) [ 2947.226541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.230847] Lustre: *** cfs_fail_loc=197, val=0*** [ 2964.832820] LustreError: 103223:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 1: rc = -5 [ 2975.137841] LustreError: 103223:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 4: rc = -110 [ 2981.308432] Lustre: *** cfs_fail_loc=197, val=0*** [ 2981.314298] Lustre: Skipped 65 previous similar messages [ 3004.959274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.962068] Lustre: Skipped 4 previous similar messages [ 3009.337792] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3016.728543] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.213542] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 15:20:40 (1776194440) [ 3050.303839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.171367] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3066.050681] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3069.849113] LustreError: 104881:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3069.859339] LustreError: 104881:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3069.871959] LustreError: 104881:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3076.066051] Lustre: 105573:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776194483/real 1776194483] req@00000000f3e86958 x1862473619777088/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776194489 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3092.950337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3111.063605] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3121.007231] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3132.315961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3139.555449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.464808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3152.192487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3159.155502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3159.244968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.530399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3159.588462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3159.722775] Lustre: lustre-MDT0000: new disk, initializing [ 3159.880750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3164.225830] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3175.359360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3175.435271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.527887] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3175.576139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.594313] LustreError: Skipped 8 previous similar messages [ 3175.746850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3180.848485] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3190.340679] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3195.504823] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3201.748590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3201.853629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3201.866194] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3206.478042] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3217.145832] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3221.703077] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3221.935896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3228.061490] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3228.406771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3232.233812] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3232.236573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.243576] LustreError: Skipped 2 previous similar messages [ 3232.245232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.262675] Lustre: Skipped 4 previous similar messages [ 3232.280319] Lustre: Skipped 8 previous similar messages [ 3237.294422] Lustre: server umount lustre-OST0000 complete [ 3237.296384] Lustre: Skipped 5 previous similar messages [ 3253.549378] LustreError: 109202:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194666 with bad export cookie 10380843701293820663 [ 3253.552144] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.557677] LustreError: 109202:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3263.689316] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 15:24:35 (1776194675) [ 3276.267223] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 3280.116568] Key type lgssc unregistered [ 3280.417875] LNet: 113890:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.427417] LNet: Removed LNI 192.168.202.115@tcp [ 3281.282716] Key type .llcrypt unregistered [ 3281.286822] Key type ._llcrypt unregistered [ 3300.574359] alg: No test for adler32 (adler32-zlib) [ 3301.338284] Key type ._llcrypt registered [ 3301.345612] Key type .llcrypt registered [ 3301.512314] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3317.047857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3318.008426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3318.802072] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3318.970920] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 3318.973543] LNet: Accept secure, port 988 [ 3320.631769] Key type lgssc registered [ 3321.692955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3329.964577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3336.621434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3342.437536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3348.505255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3353.450954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.952399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.037035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.345408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3361.390047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3361.455437] Lustre: lustre-MDT0000: new disk, initializing [ 3361.545486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.567449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3366.472780] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3375.280901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3375.343927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.349916] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3375.498303] Lustre: lustre-OST0000: new disk, initializing [ 3375.501912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3375.508705] Lustre: Skipped 1 previous similar message [ 3375.552272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.724538] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3390.007245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3390.061595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.066469] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3390.179275] Lustre: lustre-OST0001: new disk, initializing [ 3390.185789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3390.272755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.217727] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3406.824958] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.831970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.845334] Lustre: Skipped 1 previous similar message [ 3406.873286] Lustre: Skipped 1 previous similar message [ 3410.286833] Lustre: server umount lustre-MDT0000 complete [ 3417.223756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3419.107829] Lustre: 115436:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776194825/real 1776194825] req@00000000d1672d45 x1862475167313280/t0(0) o400->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776194832 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3419.149956] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3421.927029] Lustre: server umount lustre-OST0000 complete [ 3425.529415] Lustre: server umount lustre-OST0001 complete [ 3434.064934] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 15:27:25 (1776194845) [ 3442.679778] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3451.964300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3453.282959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.851305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.990416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3468.777526] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 15:28:00 (1776194880) [ 3482.661456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.072541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.989092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3488.961376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3489.846916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.724648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.990555] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 3525.186387] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3537.032458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.869610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.340149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.836681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.654201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.809430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.479425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3569.525304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3569.689631] Lustre: lustre-MDT0000: new disk, initializing [ 3569.901757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.939786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3576.681627] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3589.510650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3589.593600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.681938] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3589.738544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3589.747284] Lustre: Skipped 1 previous similar message [ 3589.877494] Lustre: lustre-MDT0001: new disk, initializing [ 3590.031362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.083912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3590.099163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3595.326697] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3606.051844] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3613.647799] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3620.832298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3620.918533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.925419] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3621.206173] Lustre: lustre-OST0000: new disk, initializing [ 3621.209843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3621.287301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.841264] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3630.225993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.236169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3637.133278] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3642.038231] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3642.320471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3646.984455] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3647.226070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3651.045642] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.046802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.052578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.060884] LustreError: Skipped 1 previous similar message [ 3651.094021] Lustre: Skipped 1 previous similar message [ 3655.651706] Lustre: server umount lustre-OST0000 complete [ 3661.281771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.283578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.306029] Lustre: Skipped 1 previous similar message [ 3661.318175] Lustre: Skipped 2 previous similar messages [ 3666.014879] Lustre: server umount lustre-MDT0000 complete [ 3666.401482] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.427284] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.455794] LustreError: Skipped 2 previous similar messages [ 3670.265572] LustreError: 128055:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195083 with bad export cookie 16700405671996002067 [ 3670.272481] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.292886] LustreError: 128055:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3679.359516] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 15:31:31 (1776195091) [ 3686.557911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.135969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.193057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.513817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.517327] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3697.647309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.951324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.536429] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3717.687662] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3724.761106] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3733.141573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.633710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.636560] Lustre: Skipped 1 previous similar message [ 3739.401451] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3747.620037] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3755.099219] Lustre: *** cfs_fail_loc=159, val=0*** [ 3761.550886] Lustre: lustre-MDT0000: Client c9bd78e3-9531-46c5-9b38-3d3571273de9 (at 192.168.202.15@tcp) reconnecting [ 3762.603221] Lustre: *** cfs_fail_loc=159, val=0*** [ 3762.607039] Lustre: Skipped 5 previous similar messages [ 3769.762983] Lustre: lustre-MDT0000: Client c9bd78e3-9531-46c5-9b38-3d3571273de9 (at 192.168.202.15@tcp) reconnecting [ 3774.943982] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3774.953349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.968778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3777.145599] Lustre: server umount lustre-OST0000 complete [ 3777.149961] Lustre: Skipped 1 previous similar message [ 3784.162991] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3784.165041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.179161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.203119] Lustre: Skipped 1 previous similar message [ 3786.207153] Lustre: 115433:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195192/real 1776195192] req@00000000ea4ca969 x1862475167351360/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776195199 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3791.327542] Lustre: 115434:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195197/real 1776195197] req@000000000353d023 x1862475167352384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776195204 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3791.333585] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.361614] LustreError: Skipped 1 previous similar message [ 3792.592755] LustreError: 130509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195205 with bad export cookie 16700405671996002921 [ 3792.607469] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.608722] LustreError: 130509:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3805.543678] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 3809.383736] Key type lgssc unregistered [ 3809.913793] LNet: 134379:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3809.923140] LNet: Removed LNI 192.168.202.115@tcp [ 3811.108311] Key type .llcrypt unregistered [ 3811.116171] Key type ._llcrypt unregistered [ 3842.761226] alg: No test for adler32 (adler32-zlib) [ 3843.520692] Key type ._llcrypt registered [ 3843.528036] Key type .llcrypt registered [ 3843.729351] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 3845.659130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3846.364531] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3846.601223] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 3846.609170] LNet: Accept secure, port 988 [ 3848.335146] Key type lgssc registered [ 3849.498692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3858.574787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.170473] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3860.234721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.592995] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.840927] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3870.690607] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.694469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.202916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.543410] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3888.431907] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3894.862580] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3902.642189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.984178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.585461] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 3911.225465] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3917.132077] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3926.354326] Lustre: *** cfs_fail_loc=159, val=0*** [ 3932.537053] Lustre: lustre-MDT0000: Client f0fb12c8-2571-4af2-a57f-c3eacdead7d1 (at 192.168.202.15@tcp) reconnecting [ 3934.059323] Lustre: *** cfs_fail_loc=159, val=0*** [ 3941.269381] Lustre: lustre-MDT0001: Client f0fb12c8-2571-4af2-a57f-c3eacdead7d1 (at 192.168.202.15@tcp) reconnecting [ 3942.460827] Lustre: *** cfs_fail_loc=159, val=0*** [ 3942.469928] Lustre: Skipped 3 previous similar messages [ 3949.968933] Lustre: lustre-MDT0001: Client f0fb12c8-2571-4af2-a57f-c3eacdead7d1 (at 192.168.202.15@tcp) reconnecting [ 3957.531371] Lustre: *** cfs_fail_loc=159, val=0*** [ 3957.535100] Lustre: Skipped 4 previous similar messages [ 3964.799692] Lustre: lustre-MDT0000: Client 6f2ea1b5-f06d-43b9-b23c-f12174bbc0fd (at 192.168.202.15@tcp) reconnecting [ 3966.118391] Lustre: *** cfs_fail_loc=159, val=0*** [ 3966.121261] Lustre: Skipped 14 previous similar messages [ 3973.521887] Lustre: lustre-MDT0000: Client 6f2ea1b5-f06d-43b9-b23c-f12174bbc0fd (at 192.168.202.15@tcp) reconnecting [ 3980.777614] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3980.778759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.796541] LustreError: Skipped 2 previous similar messages [ 3980.838955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3981.794495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3981.815252] LustreError: Skipped 5 previous similar messages [ 3981.843022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.863907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.946682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.967082] Lustre: Skipped 3 previous similar messages [ 3993.083384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.095563] Lustre: Skipped 1 previous similar message [ 3994.079559] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3994.364663] Lustre: server umount lustre-OST0000 complete [ 4003.297579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.302357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.330394] Lustre: Skipped 1 previous similar message [ 4005.402073] Lustre: server umount lustre-MDT0000 complete [ 4008.416490] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4008.439248] LustreError: Skipped 1 previous similar message [ 4009.541317] LustreError: 135848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195422 with bad export cookie 12155671940314851776 [ 4009.545846] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.550417] LustreError: 135848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4009.951629] Lustre: server umount lustre-MDT0001 complete [ 4022.622887] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4025.748861] Key type lgssc unregistered [ 4026.063531] LNet: 140259:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4026.080946] LNet: Removed LNI 192.168.202.115@tcp [ 4026.906213] Key type .llcrypt unregistered [ 4026.908848] Key type ._llcrypt unregistered [ 4048.377469] alg: No test for adler32 (adler32-zlib) [ 4049.133601] Key type ._llcrypt registered [ 4049.135691] Key type .llcrypt registered [ 4049.254199] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4050.524891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4051.011206] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4051.203170] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4051.206586] LNet: Accept secure, port 988 [ 4052.863155] Key type lgssc registered [ 4054.125547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4063.032218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.345793] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.411833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.533775] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4069.858082] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.978663] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.439864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.709494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.788382] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4092.429472] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4098.944473] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4106.912230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.371198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.423106] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 4112.582663] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4114.544397] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 4121.066194] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4132.577848] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 4133.630785] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 4137.702967] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 4144.610092] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4144.618325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.650840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.632969] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.634352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.649899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.665895] Lustre: Skipped 1 previous similar message [ 4149.471226] Lustre: server umount lustre-OST0000 complete [ 4154.336449] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4154.347922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.379275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4160.994106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4161.002403] Lustre: Skipped 3 previous similar messages [ 4166.123302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4166.132683] Lustre: Skipped 1 previous similar message [ 4168.160072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4168.455290] Lustre: server umount lustre-MDT0000 complete [ 4171.232255] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.246524] LustreError: Skipped 1 previous similar message [ 4172.775975] LustreError: 141707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195585 with bad export cookie 374596079846177544 [ 4172.786512] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4173.042694] Lustre: server umount lustre-MDT0001 complete [ 4186.177891] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4189.426972] Key type lgssc unregistered [ 4189.591873] LNet: 145790:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4189.606388] LNet: Removed LNI 192.168.202.115@tcp [ 4190.265521] Key type .llcrypt unregistered [ 4190.267810] Key type ._llcrypt unregistered [ 4217.027676] alg: No test for adler32 (adler32-zlib) [ 4217.785448] Key type ._llcrypt registered [ 4217.786649] Key type .llcrypt registered [ 4217.935266] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4219.275043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4220.419198] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4220.786504] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4220.811618] LNet: Accept secure, port 988 [ 4222.535104] Key type lgssc registered [ 4223.644547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4231.607928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.326560] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.442482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.822662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.525601] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4243.937987] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.046766] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.263957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.522978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4255.360051] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4262.941656] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4268.791145] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4275.429156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4275.775875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.863060] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 4280.362530] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4283.967248] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 4287.144146] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4292.590552] Lustre: *** cfs_fail_loc=159, val=0*** [ 4300.166623] Lustre: lustre-MDT0000: Client cd990b50-4623-4d3a-8f7e-16aa7d359ece (at 192.168.202.15@tcp) reconnecting [ 4304.352982] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4304.357832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.365406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4305.384100] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4305.391276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.400082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.426408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.434827] Lustre: Skipped 1 previous similar message [ 4309.674902] Lustre: server umount lustre-OST0000 complete [ 4316.128223] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4316.143302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.176811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4320.094736] Lustre: server umount lustre-MDT0000 complete [ 4322.784824] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.821277] LustreError: Skipped 1 previous similar message [ 4325.303180] LustreError: 150237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195738 with bad export cookie 9337724948518763754 [ 4325.307198] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.318168] LustreError: 150237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4325.792967] Lustre: server umount lustre-MDT0001 complete [ 4339.904460] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4343.189710] Key type lgssc unregistered [ 4343.619773] LNet: 151041:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4343.627992] LNet: Removed LNI 192.168.202.115@tcp [ 4344.766406] Key type .llcrypt unregistered [ 4344.768766] Key type ._llcrypt unregistered [ 4369.728257] alg: No test for adler32 (adler32-zlib) [ 4370.487197] Key type ._llcrypt registered [ 4370.491027] Key type .llcrypt registered [ 4370.606591] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4371.995398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.956635] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4373.128764] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4373.131552] LNet: Accept secure, port 988 [ 4374.783183] Key type lgssc registered [ 4376.242426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4385.595817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.277285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.339750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.419152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.987733] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4397.546562] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.234421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.833084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.236894] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4415.606074] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4422.387314] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4429.298454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.790235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.879503] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 4435.257283] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4436.987300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 4445.550715] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4465.459810] Lustre: lustre-OST0000: evicting 72281f67-554f-4406-a601-57672e223dda (at 192.168.202.15@tcp) by administrative request [ 4465.472348] Lustre: Skipped 1 previous similar message [ 4498.402904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.408922] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4498.428805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4498.446316] Lustre: Skipped 1 previous similar message [ 4500.296938] Lustre: server umount lustre-OST0000 complete [ 4508.641669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.669197] Lustre: Skipped 1 previous similar message [ 4508.672900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4508.684288] Lustre: Skipped 1 previous similar message [ 4510.722468] Lustre: server umount lustre-MDT0000 complete [ 4513.770246] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.789231] LustreError: Skipped 1 previous similar message [ 4515.104325] LustreError: 152485:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195928 with bad export cookie 3837742113357270283 [ 4515.119869] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4515.121359] LustreError: 152485:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4515.548919] Lustre: server umount lustre-MDT0001 complete [ 4528.142000] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4531.033908] Key type lgssc unregistered [ 4531.382970] LNet: 156700:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4531.388451] LNet: Removed LNI 192.168.202.115@tcp [ 4532.197456] Key type .llcrypt unregistered [ 4532.201793] Key type ._llcrypt unregistered [ 4600.919693] alg: No test for adler32 (adler32-zlib) [ 4601.679266] Key type ._llcrypt registered [ 4601.685261] Key type .llcrypt registered [ 4601.816707] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4603.098685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4603.628960] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4603.809895] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4603.817550] LNet: Accept secure, port 988 [ 4605.527144] Key type lgssc registered [ 4606.960242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4615.248198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.705535] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.800179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.044966] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4622.306748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.426970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.855897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.166782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4635.932199] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4644.120648] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.474947] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4658.292505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.781985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.878510] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 4664.649127] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4667.002535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4673.934459] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4682.722609] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4682.724092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.735172] LustreError: Skipped 1 previous similar message [ 4682.737882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4682.762448] Lustre: Skipped 1 previous similar message [ 4682.796535] Lustre: Skipped 1 previous similar message [ 4687.843598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4688.675311] Lustre: server umount lustre-OST0000 complete [ 4692.966420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.984133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.986884] Lustre: Skipped 1 previous similar message [ 4697.572924] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.584819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.605611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.965520] Lustre: server umount lustre-MDT0000 complete [ 4703.200647] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.218202] LustreError: Skipped 1 previous similar message [ 4703.636325] LustreError: 162101:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196116 with bad export cookie 13463888755589654558 [ 4703.636822] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.644757] LustreError: 162101:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4704.047552] Lustre: server umount lustre-MDT0001 complete [ 4716.838592] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4720.054389] Key type lgssc unregistered [ 4720.463631] LNet: 162904:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4720.474709] LNet: Removed LNI 192.168.202.115@tcp [ 4721.915395] Key type .llcrypt unregistered [ 4721.918543] Key type ._llcrypt unregistered [ 4749.501550] alg: No test for adler32 (adler32-zlib) [ 4750.259483] Key type ._llcrypt registered [ 4750.268628] Key type .llcrypt registered [ 4750.433776] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4751.669884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4752.222091] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4752.587824] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4752.595426] LNet: Accept secure, port 988 [ 4754.295156] Key type lgssc registered [ 4755.535290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4763.470307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4772.187224] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 15:49:44 (1776196184) [ 4791.845257] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 4801.980331] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4813.118414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4820.568595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.201790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4834.484763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4842.340224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4842.435951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.831571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4843.867839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4843.966434] Lustre: lustre-MDT0000: new disk, initializing [ 4844.095747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.118551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4849.419845] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4860.873380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.954047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.058236] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4861.104876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4861.110232] Lustre: Skipped 1 previous similar message [ 4861.213156] Lustre: lustre-MDT0001: new disk, initializing [ 4861.328242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4861.393153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4861.413658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4866.078315] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4875.692939] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4881.553960] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.167372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4888.235938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.243605] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4888.500643] Lustre: lustre-OST0000: new disk, initializing [ 4888.514557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4888.646678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.719409] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 4894.807712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4894.823903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4904.425433] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4915.681130] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.685439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.693950] LustreError: Skipped 1 previous similar message [ 4915.711767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.723639] Lustre: Skipped 1 previous similar message [ 4920.333026] Lustre: server umount lustre-OST0000 complete [ 4925.921140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.930020] Lustre: Skipped 1 previous similar message [ 4925.938456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4925.949760] Lustre: Skipped 1 previous similar message [ 4930.461319] Lustre: server umount lustre-MDT0000 complete [ 4931.041481] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.055256] LustreError: Skipped 1 previous similar message [ 4934.433970] LustreError: 167330:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196347 with bad export cookie 4458182966586779634 [ 4934.440967] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4934.441144] LustreError: 167330:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4934.791901] Lustre: server umount lustre-MDT0001 complete [ 4946.805512] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 4949.471193] Key type lgssc unregistered [ 4949.690411] LNet: 171335:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4949.703513] LNet: Removed LNI 192.168.202.115@tcp [ 4950.360136] Key type .llcrypt unregistered [ 4950.363559] Key type ._llcrypt unregistered [ 4976.630371] alg: No test for adler32 (adler32-zlib) [ 4977.394546] Key type ._llcrypt registered [ 4977.397291] Key type .llcrypt registered [ 4977.546383] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 4978.521625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.957765] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 4979.233970] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 4979.245993] LNet: Accept secure, port 988 [ 4981.039135] Key type lgssc registered [ 4982.507429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.070297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.605072] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.707977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.121153] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.161328] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5003.236887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.322790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.355039] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.689258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5014.223712] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5022.346866] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5028.391598] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5035.591328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5036.036983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.912850] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5043.181987] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5048.389668] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5052.725053] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5052.746819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.763110] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5052.766385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.773623] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 5052.777362] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 5053.699357] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5053.712745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.729281] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5053.738824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.749921] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 5053.755197] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 5054.573317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.587736] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:161 [ 5055.312229] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5055.315832] Lustre: Skipped 1 previous similar message [ 5055.330279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.339992] Lustre: Skipped 1 previous similar message [ 5055.343734] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5055.350826] Lustre: Skipped 1 previous similar message [ 5055.358332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.366273] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 5055.367018] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:193 [ 5055.372763] Lustre: Skipped 1 previous similar message [ 5056.077477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.088822] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:225 [ 5056.733351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.742629] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:257 [ 5057.546073] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5057.554512] Lustre: Skipped 2 previous similar messages [ 5057.566203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.584444] Lustre: Skipped 2 previous similar messages [ 5057.586421] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5057.593854] Lustre: Skipped 2 previous similar messages [ 5057.603095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.616405] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 5057.620613] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:289 [ 5057.622128] Lustre: Skipped 2 previous similar messages [ 5058.265477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.280677] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:321 [ 5058.926086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.941643] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:353 [ 5059.705050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.719115] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:385 [ 5067.764169] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5067.769359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.775460] LustreError: Skipped 1 previous similar message [ 5067.795351] Lustre: Skipped 3 previous similar messages [ 5067.801396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.774821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.790624] Lustre: Skipped 1 previous similar message [ 5073.778599] Lustre: server umount lustre-OST0000 complete [ 5079.010812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.018034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5079.018788] Lustre: Skipped 2 previous similar messages [ 5079.034293] Lustre: Skipped 1 previous similar message [ 5084.131101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.143226] Lustre: Skipped 1 previous similar message [ 5089.249169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.254600] Lustre: Skipped 1 previous similar message [ 5092.319237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5092.451270] Lustre: server umount lustre-MDT0000 complete [ 5094.374824] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.402126] LustreError: Skipped 1 previous similar message [ 5096.285232] LustreError: 172804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196509 with bad export cookie 13019867254644461368 [ 5096.302701] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.814534] Lustre: server umount lustre-MDT0001 complete [ 5109.268665] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 5112.440828] Key type lgssc unregistered [ 5112.578730] LNet: 177281:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5112.585516] LNet: Removed LNI 192.168.202.115@tcp [ 5113.568812] Key type .llcrypt unregistered [ 5113.573821] Key type ._llcrypt unregistered [ 5145.438374] alg: No test for adler32 (adler32-zlib) [ 5146.206712] Key type ._llcrypt registered [ 5146.208403] Key type .llcrypt registered [ 5146.408388] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5147.574199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5147.908428] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5148.092498] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 5148.095105] LNet: Accept secure, port 988 [ 5149.703092] Key type lgssc registered [ 5150.901289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5159.542842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.081676] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.197484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.312220] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5166.563282] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.691683] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.526299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.983384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.465521] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5189.571868] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5196.125906] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5202.939914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.390954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.047558] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5212.692884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:417 [ 5217.674738] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5223.126827] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5224.928613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.939950] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5224.942842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.947740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 5224.948223] LustreError: 178790:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5228.006501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.010951] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.030428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.945113] Lustre: lustre-OST0000: Not available for connect from 192.168.202.15@tcp (stopping) [ 5229.956687] Lustre: Skipped 1 previous similar message [ 5233.123533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.135127] Lustre: Skipped 1 previous similar message [ 5238.243708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5238.257188] Lustre: Skipped 2 previous similar messages [ 5241.311140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5241.640595] Lustre: server umount lustre-OST0000 complete [ 5244.959324] LustreError: 178790:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 5244.966154] LustreError: 178790:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 5276.082578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5276.459536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.419982] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5289.784186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.815082] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.817254] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) [ 5289.837547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:141 to 0x0:449 [ 5289.858186] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 5289.859337] Lustre: Skipped 1 previous similar message [ 5291.384037] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5292.993243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5299.736879] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95b9ca160800.ost_server_uuid 40 [ 5301.280757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95b9ca160800.ost_server_uuid in FULL state after 0 sec [ 5308.008976] Lustre: server umount lustre-OST0000 complete [ 5314.016793] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5314.032119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.051376] Lustre: Skipped 1 previous similar message [ 5314.057171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5314.071079] Lustre: Skipped 1 previous similar message [ 5317.599242] Lustre: 178080:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196723/real 1776196723] req@000000007710a639 x1862477085175616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776196730 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5317.599356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.634676] Lustre: 178080:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5317.677904] Lustre: Skipped 1 previous similar message [ 5319.647209] Lustre: 183265:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196726/real 1776196726] req@00000000f99c042b x1862477085176192/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196732 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5319.813960] Lustre: server umount lustre-MDT0000 complete [ 5322.724789] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.732929] LustreError: Skipped 1 previous similar message [ 5322.783403] Lustre: 178081:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196728/real 1776196728] req@00000000a431d587 x1862477085176512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196735 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5324.048300] LustreError: 178750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196737 with bad export cookie 15190734212121851766 [ 5324.058479] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5324.076364] LustreError: 178750:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5324.493732] Lustre: server umount lustre-MDT0001 complete [ 5340.761961] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 5344.589095] Key type lgssc unregistered [ 5344.861048] LNet: 184070:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5344.869808] LNet: Removed LNI 192.168.202.115@tcp [ 5346.290600] Key type .llcrypt unregistered [ 5346.292078] Key type ._llcrypt unregistered [ 5382.372625] alg: No test for adler32 (adler32-zlib) [ 5383.140902] Key type ._llcrypt registered [ 5383.143300] Key type .llcrypt registered [ 5383.292094] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5384.664618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5385.415928] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5385.668985] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 5385.672851] LNet: Accept secure, port 988 [ 5387.383164] Key type lgssc registered [ 5388.472858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5397.300275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.780487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.853139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5403.754278] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5404.131946] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.250686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.688358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.199321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5418.761560] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5426.926890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.313616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.417079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 5432.007608] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5440.170124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5440.257755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5440.267169] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5440.395607] Lustre: lustre-OST0001: new disk, initializing [ 5440.400223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5440.486636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5444.649361] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5450.312041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5450.325524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5456.181068] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5461.967862] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5486.049450] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5486.085501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.106865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.684113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.693481] Lustre: Skipped 6 previous similar messages [ 5496.288429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5496.453810] Lustre: server umount lustre-MDT0000 complete [ 5496.817700] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5501.363842] LustreError: 188193:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196914 with bad export cookie 12647140421684172682 [ 5501.383596] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5501.812262] Lustre: server umount lustre-MDT0001 complete [ 5508.002213] Lustre: 185138:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196914/real 1776196914] req@00000000b858a503 x1862477333705472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196921 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 5508.043167] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.051224] Lustre: Skipped 3 previous similar messages [ 5512.671702] Lustre: 189670:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196919/real 1776196919] req@0000000074afd218 x1862477333705600/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196925 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5512.799830] Lustre: server umount lustre-OST0000 complete [ 5514.211372] Lustre: 185137:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196920/real 1776196920] req@00000000af2a1e82 x1862477333705728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776196927 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 5518.636300] Lustre: server umount lustre-OST0001 complete [ 5522.663043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5522.786347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5523.957298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5524.078984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5525.387468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5525.537343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5526.892192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5527.032843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5543.061540] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5555.301446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.961515] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.986467] LustreError: Skipped 3 previous similar messages [ 5556.067191] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.005511] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5566.433404] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.457375] LustreError: Skipped 2 previous similar messages [ 5572.502316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.852668] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5578.294623] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5588.895674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5589.219153] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5594.169702] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5602.197318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.465602] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 5606.473716] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 5606.945670] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5616.297399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5620.539894] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5637.092847] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.096935] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.123280] Lustre: Skipped 1 previous similar message [ 5637.137706] Lustre: Skipped 3 previous similar messages [ 5642.209082] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.227574] Lustre: Skipped 3 previous similar messages [ 5642.804628] Lustre: server umount mylustre-MDT0000 complete [ 5646.789659] LustreError: 191313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197059 with bad export cookie 12647140421684175405 [ 5646.798484] LustreError: 191313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5646.810076] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5654.367947] Lustre: 185136:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197060/real 1776197060] req@00000000c603d248 x1862477333736640/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197067 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5654.389981] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.404080] Lustre: Skipped 2 previous similar messages [ 5657.662441] Lustre: server umount mylustre-OST0000 complete [ 5657.669466] Lustre: Skipped 1 previous similar message [ 5658.596086] Lustre: 185137:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197064/real 1776197064] req@00000000da54a84f x1862477333736832/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197071 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 5658.617595] Lustre: 185137:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5665.531173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5665.640864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5666.742814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5666.883045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5667.999837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5668.168309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5669.357179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5669.456964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.161728] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5694.204528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.669454] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.686284] LustreError: Skipped 1 previous similar message [ 5694.757308] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.761658] Lustre: Skipped 1 previous similar message [ 5700.288922] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5712.829193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.923910] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5728.537848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5728.613259] LustreError: 137-5: tfs-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5728.621350] LustreError: Skipped 3 previous similar messages [ 5733.970919] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5743.649794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5746.450852] Lustre: tfs-OST0000: deleting orphan objects from 0x0:483 to 0x0:545 [ 5746.467074] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 5748.837904] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5756.689430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5760.646191] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5774.817328] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5774.832776] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.847187] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5775.136475] Lustre: server umount tfs-MDT0000 complete [ 5775.139546] Lustre: Skipped 1 previous similar message [ 5778.985212] LustreError: 196681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197192 with bad export cookie 12647140421684177239 [ 5778.994772] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.001644] LustreError: 196681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5789.599175] Lustre: 185136:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197195/real 1776197195] req@00000000078e4066 x1862477333762624/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197202 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 5789.638051] Lustre: 185136:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5789.653246] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.669709] Lustre: Skipped 3 previous similar messages [ 5797.232619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5797.325299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.630906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5798.729428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5799.754590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5799.848033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5801.202931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5801.278773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5816.182281] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 5826.531122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.048592] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.070354] LustreError: Skipped 6 previous similar messages [ 5827.181429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.183898] Lustre: Skipped 3 previous similar messages [ 5832.252822] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5842.502984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.822330] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5858.775957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5860.355285] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:577 [ 5865.005211] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5876.065942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.982430] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 5881.958228] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 5890.348780] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5894.462057] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5906.906317] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 16:08:38 (1776197318) [ 5924.839173] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5924.848146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.873321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.884337] Lustre: server umount lustre-MDT0000 complete [ 5929.888401] Lustre: Skipped 3 previous similar messages [ 5933.950753] LustreError: 201989:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197347 with bad export cookie 12647140421684178870 [ 5933.959755] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.971288] LustreError: 201989:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5945.322348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5945.340238] Lustre: Skipped 9 previous similar messages [ 5948.383171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5967.327812] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5983.031502] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 5991.300983] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6003.843504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6012.323544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.523853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.897955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6036.282372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6036.361599] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6036.361599] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6036.361599] [ 6036.372929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6036.578688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6036.641522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6036.740154] Lustre: lustre-MDT0000: new disk, initializing [ 6036.793928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6036.796586] Lustre: Skipped 3 previous similar messages [ 6036.810085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6043.494792] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6058.891186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6058.949725] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 6058.949725] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6058.949725] [ 6058.962626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6059.048198] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6059.115254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6059.122227] Lustre: Skipped 1 previous similar message [ 6059.191939] Lustre: lustre-MDT0001: new disk, initializing [ 6059.345553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6059.355656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6065.347457] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6076.541835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.616867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.622677] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6077.042462] Lustre: lustre-OST0000: new disk, initializing [ 6077.056810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6078.419141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6078.432172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6082.415276] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6092.550477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6092.614912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6092.619675] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6092.706493] Lustre: lustre-OST0001: new disk, initializing [ 6092.710060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6093.867419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6093.876900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6097.596064] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6108.131418] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 16:11:59 (1776197519) [ 6129.174669] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 16:12:20 (1776197540) [ 6144.999473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.000409] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.003296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.019210] Lustre: Skipped 6 previous similar messages [ 6145.024316] LustreError: Skipped 1 previous similar message [ 6150.399396] Lustre: server umount lustre-OST0000 complete [ 6150.409143] Lustre: Skipped 3 previous similar messages [ 6155.233338] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.248282] LustreError: Skipped 20 previous similar messages [ 6164.901165] LustreError: 208277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197577 with bad export cookie 12647140421684180389 [ 6164.904581] LustreError: 208277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6164.907826] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.295168] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6198.788198] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6218.020152] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6232.637350] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6235.235895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6242.780948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6249.134825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.849150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6261.927669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6268.728571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6268.823248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.212478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6269.239242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6269.351918] Lustre: lustre-MDT0000: new disk, initializing [ 6269.479370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6274.895193] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6285.987374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.089110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.226434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6286.231206] Lustre: Skipped 1 previous similar message [ 6286.325942] Lustre: lustre-MDT0001: new disk, initializing [ 6286.540044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6286.558535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6291.446554] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6301.105090] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6307.413994] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6314.288715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.358308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.363263] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6314.611460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.614978] Lustre: Skipped 5 previous similar messages [ 6316.412612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6319.190158] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6328.404936] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6337.000415] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.003749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.016788] LustreError: Skipped 1 previous similar message [ 6337.017509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.032198] Lustre: Skipped 5 previous similar messages [ 6337.044691] Lustre: Skipped 14 previous similar messages [ 6341.230551] Lustre: server umount lustre-OST0000 complete [ 6341.232923] Lustre: Skipped 3 previous similar messages [ 6354.314700] LustreError: 217615:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197767 with bad export cookie 12647140421684181817 [ 6354.322132] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6354.330433] LustreError: 217615:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6366.169356] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6369.736795] Key type lgssc unregistered [ 6370.073807] LNet: 219377:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6370.081327] LNet: Removed LNI 192.168.202.115@tcp [ 6371.155337] Key type .llcrypt unregistered [ 6371.159250] Key type ._llcrypt unregistered [ 6395.795629] alg: No test for adler32 (adler32-zlib) [ 6396.552800] Key type ._llcrypt registered [ 6396.563857] Key type .llcrypt registered [ 6396.702821] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6397.971564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6398.323207] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6398.654196] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6398.661877] LNet: Accept secure, port 988 [ 6400.327115] Key type lgssc registered [ 6401.144604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6408.867432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.425681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.591850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.483601] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6415.847513] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.965952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.468862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.753787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6430.924974] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6439.190746] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6446.381766] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6453.728604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.201495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.843332] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6460.364309] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 6468.553713] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6470.057133] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 6475.744976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.752708] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.784283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.803344] Lustre: Skipped 1 previous similar message [ 6478.801673] Lustre: server umount lustre-OST0000 complete [ 6485.991367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.997471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.005107] Lustre: Skipped 2 previous similar messages [ 6486.015699] Lustre: Skipped 1 previous similar message [ 6491.108558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6491.114631] Lustre: Skipped 1 previous similar message [ 6496.227048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.234820] Lustre: Skipped 1 previous similar message [ 6497.248749] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6497.383142] Lustre: server umount lustre-MDT0000 complete [ 6501.345408] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.353641] LustreError: Skipped 1 previous similar message [ 6502.013034] LustreError: 220869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197915 with bad export cookie 6301216474382008950 [ 6502.014092] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.027553] LustreError: 220869:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6502.296425] Lustre: server umount lustre-MDT0001 complete [ 6515.329599] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6518.439648] Key type lgssc unregistered [ 6518.807350] LNet: 224602:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6518.818452] LNet: Removed LNI 192.168.202.115@tcp [ 6520.113338] Key type .llcrypt unregistered [ 6520.118478] Key type ._llcrypt unregistered [ 6541.902706] alg: No test for adler32 (adler32-zlib) [ 6542.672422] Key type ._llcrypt registered [ 6542.676834] Key type .llcrypt registered [ 6542.827737] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6544.937771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6545.657133] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6545.877911] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6545.885342] LNet: Accept secure, port 988 [ 6547.655151] Key type lgssc registered [ 6548.505936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6559.820340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.503312] LustreError: 225868:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 1: rc = -5 [ 6608.223253] LustreError: 225868:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.115@tcp: failed processing log, type 4: rc = -110 [ 6638.943537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.570550] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6650.876911] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6652.846133] Lustre: server umount lustre-OST0000 complete [ 6674.985980] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 6680.302598] Key type lgssc unregistered [ 6680.745285] LNet: 227515:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6680.762125] LNet: Removed LNI 192.168.202.115@tcp [ 6681.969296] Key type .llcrypt unregistered [ 6681.972926] Key type ._llcrypt unregistered [ 6720.065916] alg: No test for adler32 (adler32-zlib) [ 6720.827970] Key type ._llcrypt registered [ 6720.830306] Key type .llcrypt registered [ 6720.998226] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6722.543425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6723.006805] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6723.227849] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 6723.232870] LNet: Accept secure, port 988 [ 6724.895164] Key type lgssc registered [ 6725.914524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6738.126906] loop0: detected capacity change from 0 to 204800000 [ 6738.279729] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.321742] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.686779] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.699778] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.758884] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.884955] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6738.945102] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6739.073737] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6739.176978] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6739.305347] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6740.695264] loop0: detected capacity change from 0 to 204800000 [ 6740.901844] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6741.424522] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6742.507287] loop0: detected capacity change from 0 to 204800000 [ 6743.033753] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6743.990193] loop0: detected capacity change from 0 to 204800000 [ 6744.122468] print_req_error: 12 callbacks suppressed [ 6744.122483] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6744.147448] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6744.248223] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6744.263571] blk_update_request: operation not supported error, dev loop0, sector 58272 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6744.391797] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6745.389782] loop0: detected capacity change from 0 to 204800000 [ 6745.427357] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6749.378943] loop0: detected capacity change from 0 to 204800000 [ 6749.431691] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6753.130525] loop0: detected capacity change from 0 to 204800000 [ 6753.164968] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6757.739244] loop0: detected capacity change from 0 to 204800000 [ 6757.790995] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6761.671581] loop0: detected capacity change from 0 to 204800000 [ 6761.744163] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6761.788428] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.841626] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6761.847147] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6761.854753] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6761.862886] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6761.871153] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6761.878484] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6761.885233] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6761.891977] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6761.901684] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6761.914412] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6761.926635] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6761.938763] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6761.954339] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6761.967819] Lustre: 230977:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6772.191161] Lustre: 231256:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198179/real 1776198179] req@0000000085a168ba x1862478736654464/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198185 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6772.244627] Lustre: server umount MGS complete [ 6773.091345] loop0: detected capacity change from 0 to 204800000 [ 6773.209276] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.578886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.635208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.680853] LustreError: 231310:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 6774.584138] loop1: detected capacity change from 0 to 204800000 [ 6774.704845] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6774.816898] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.876502] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6774.883520] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6774.892300] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6774.906765] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6774.920278] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6774.945228] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6774.958337] Lustre: 231424:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6774.983321] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6775.246752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6775.356075] LustreError: 231424:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0001: Aborting client recovery [ 6776.152817] loop2: detected capacity change from 0 to 204800000 [ 6776.336302] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6776.443090] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.458548] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 6776.558177] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6776.574409] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6776.590140] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6776.616985] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6776.627962] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6776.654331] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6776.668093] Lustre: 231507:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6776.840780] Lustre: Found index 0 for lustre-OST0000, updating log [ 6776.925676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.948373] LustreError: 231367:0:(lod_dev.c:528:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 4, retries 0, failed: rc = -108 [ 6776.960838] LustreError: 231507:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 6777.949311] loop3: detected capacity change from 0 to 204800000 [ 6778.144216] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6778.349489] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.352803] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 6778.398799] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6778.415649] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6778.438450] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6778.456220] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6778.475672] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6778.496610] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6778.515304] Lustre: 231594:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6778.557806] Lustre: Found index 1 for lustre-OST0001, updating log [ 6778.786548] LustreError: 231594:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 6779.770464] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 6779.798524] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 6780.783156] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 6780.794880] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 6787.854581] LustreError: 231845:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1152us: rc = -2 [ 6788.367501] LustreError: 231332:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 2505us: rc = -2 [ 6788.401044] LustreError: 231332:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 26 previous similar messages [ 6801.377473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.397838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.399358] Lustre: Skipped 2 previous similar messages [ 6801.422395] Lustre: Skipped 3 previous similar messages [ 6806.507648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6806.524757] Lustre: Skipped 3 previous similar messages [ 6807.376850] Lustre: server umount lustre-MDT0000 complete [ 6808.500945] LustreError: 231319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198221 with bad export cookie 14983735948892324067 [ 6808.517092] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.616704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.616993] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.617633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6811.627929] LustreError: Skipped 1 previous similar message [ 6811.653857] Lustre: Skipped 2 previous similar messages [ 6813.664208] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6813.664568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6813.677941] LustreError: Skipped 1 previous similar message [ 6813.701016] Lustre: Skipped 2 previous similar messages [ 6815.137278] Lustre: server umount lustre-MDT0001 complete [ 6822.478267] Lustre: server umount lustre-OST0000 complete [ 6838.239511] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6838.372735] Lustre: server umount lustre-OST0001 complete [ 6846.914649] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 16:24:19 (1776198259) [ 6872.143658] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 6880.693305] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 6889.452125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6894.918929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6900.152247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6907.957601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6916.462474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6916.549749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.836592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6916.876615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6916.990635] Lustre: lustre-MDT0000: new disk, initializing [ 6917.174867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.182511] Lustre: Skipped 1 previous similar message [ 6917.236035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6922.568146] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6932.724714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.789344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.864810] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6932.926935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6932.929926] Lustre: Skipped 1 previous similar message [ 6932.994222] Lustre: lustre-MDT0001: new disk, initializing [ 6933.068057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.103214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6933.107361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6937.872513] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6946.760618] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6951.792653] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6957.612500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6957.672826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.681188] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6957.833866] Lustre: lustre-OST0000: new disk, initializing [ 6957.837253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6957.920586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.325539] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 6967.902815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6967.912567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6972.065783] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6977.723858] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6990.107394] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6998.892167] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 7050.210103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.221819] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.247179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7050.247190] Lustre: Skipped 2 previous similar messages [ 7053.951980] Lustre: server umount lustre-OST0000 complete [ 7060.453869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.460168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.467463] Lustre: Skipped 2 previous similar messages [ 7060.476656] Lustre: Skipped 2 previous similar messages [ 7065.569983] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7065.585374] LustreError: Skipped 3 previous similar messages [ 7067.583261] LustreError: 235769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198480 with bad export cookie 14983735948892326398 [ 7067.586088] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.594221] LustreError: 235769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7076.258313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.037752] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7092.192122] Lustre: 241317:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198499/real 1776198499] req@00000000c27e96a3 x1862478736726656/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198505 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 7092.276938] Lustre: server umount MGS complete [ 7092.282803] Lustre: Skipped 2 previous similar messages [ 7105.246765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.856212] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.866578] LustreError: Skipped 2 previous similar messages [ 7105.899974] LustreError: 241953:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7105.983982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7110.958990] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7116.262096] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.271385] LustreError: Skipped 1 previous similar message [ 7119.227980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.076127] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7131.592474] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7137.222265] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7143.927723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7144.391127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.396886] Lustre: Skipped 1 previous similar message [ 7149.716857] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7159.416177] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7190.501089] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7190.512784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.533563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.562699] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.387743] Lustre: server umount lustre-OST0000 complete [ 7201.762052] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.773810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.791933] Lustre: Skipped 1 previous similar message [ 7207.905039] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.925447] LustreError: Skipped 1 previous similar message [ 7208.378395] LustreError: 244737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198621 with bad export cookie 14983735948892328309 [ 7208.389237] LustreError: 244737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7208.392475] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7220.096758] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7222.434775] Key type lgssc unregistered [ 7222.724780] LNet: 246060:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.749802] LNet: Removed LNI 192.168.202.115@tcp [ 7223.544818] Key type .llcrypt unregistered [ 7223.546426] Key type ._llcrypt unregistered [ 7257.886935] alg: No test for adler32 (adler32-zlib) [ 7258.642409] Key type ._llcrypt registered [ 7258.643665] Key type .llcrypt registered [ 7258.720321] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7271.551303] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7273.148477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7273.582267] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7273.771733] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7273.773936] LNet: Accept secure, port 988 [ 7275.455141] Key type lgssc registered [ 7276.727377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7286.386754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7293.008550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7299.012295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7304.768468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.312237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.443430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.704808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7312.727593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7312.796456] Lustre: lustre-MDT0000: new disk, initializing [ 7312.859142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.871765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7316.797436] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7325.740283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7325.829558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.881905] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7325.907738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7325.910482] Lustre: Skipped 1 previous similar message [ 7325.968158] Lustre: lustre-MDT0001: new disk, initializing [ 7326.030882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7326.050934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7326.056436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7331.242691] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7340.829339] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7345.669916] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7351.421489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7351.505588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.511787] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7351.770305] Lustre: lustre-OST0000: new disk, initializing [ 7351.773319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7351.825455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7355.833963] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7357.502953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7357.519868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7364.367696] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7370.332794] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7378.880264] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7386.328834] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 7429.602606] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7429.650426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.665798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7429.684369] Lustre: Skipped 1 previous similar message [ 7434.722509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.725652] Lustre: Skipped 2 previous similar messages [ 7439.846577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7439.859191] Lustre: Skipped 1 previous similar message [ 7441.375163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7441.555847] Lustre: server umount lustre-OST0000 complete [ 7449.056621] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7449.067206] LustreError: Skipped 1 previous similar message [ 7449.071794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.086485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.096070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.203635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.211312] Lustre: Skipped 3 previous similar messages [ 7460.319467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7460.401529] Lustre: server umount lustre-MDT0000 complete [ 7464.071164] LustreError: 250997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198877 with bad export cookie 15259303321575741410 [ 7464.085635] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7464.094453] LustreError: 250997:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7464.577361] Lustre: server umount lustre-MDT0001 complete [ 7474.376378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.643820] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7489.503241] Lustre: 255536:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198896/real 1776198896] req@00000000a0526ea3 x1862479313418560/t0(0) o251->MGC192.168.202.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198902 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7489.560616] Lustre: server umount MGS complete [ 7499.180694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.552694] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.570887] LustreError: 256171:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7499.624713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.138996] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7504.872350] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7509.404319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.733275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7513.736635] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7520.368868] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7525.187446] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7531.035149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.312412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.331156] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7542.075693] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7585.763772] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7585.764755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.768046] LustreError: Skipped 1 previous similar message [ 7585.776607] Lustre: Skipped 1 previous similar message [ 7585.776904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7585.791806] Lustre: Skipped 2 previous similar messages [ 7589.391702] Lustre: server umount lustre-OST0000 complete [ 7596.001199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.014135] Lustre: Skipped 1 previous similar message [ 7599.300950] Lustre: server umount lustre-MDT0000 complete [ 7601.124403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.140302] LustreError: Skipped 1 previous similar message [ 7602.871978] LustreError: 256139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199015 with bad export cookie 15259303321575743209 [ 7602.872328] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7602.882153] LustreError: 256139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7615.309418] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7618.436182] Key type lgssc unregistered [ 7618.697935] LNet: 260299:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7618.723957] LNet: Removed LNI 192.168.202.115@tcp [ 7619.462672] Key type .llcrypt unregistered [ 7619.467678] Key type ._llcrypt unregistered [ 7644.013579] alg: No test for adler32 (adler32-zlib) [ 7644.766468] Key type ._llcrypt registered [ 7644.772475] Key type .llcrypt registered [ 7644.886883] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7645.903442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.611504] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7646.802285] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7646.806353] LNet: Accept secure, port 988 [ 7648.471131] Key type lgssc registered [ 7649.492857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7657.679928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.196432] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.261847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.033458] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7664.613892] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.743689] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.778477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.307274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.358162] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7687.941708] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7694.401710] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7702.575937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.373953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.024823] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7718.607460] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7725.780755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7725.866361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7725.872092] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7725.992473] Lustre: lustre-OST0001: new disk, initializing [ 7725.997608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7726.075068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.284781] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 7731.298630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7731.313040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7731.365393] LustreError: 265086:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7732.452683] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7732.458205] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7733.473714] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7733.488756] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 7735.535749] LustreError: 265086:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7735.546670] LustreError: 265086:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7739.826821] LustreError: 265086:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7739.843486] LustreError: 265086:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 7743.485489] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7743.684505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7748.067068] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7748.082937] LustreError: 265092:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 7749.496852] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95b9c4b6e800.ost_server_uuid 40 [ 7751.072966] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95b9c4b6e800.ost_server_uuid in FULL state after 0 sec [ 7756.545830] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95b9c4b6e800.ost_server_uuid 40 [ 7758.377117] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff95b9c4b6e800.ost_server_uuid in FULL state after 0 sec [ 7774.066940] Lustre: lustre-OST0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 7775.714623] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7775.727143] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7775.748989] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7776.231397] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7776.244591] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.761094] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7777.765959] Lustre: Skipped 1 previous similar message [ 7779.664586] Lustre: server umount lustre-OST0001 complete [ 7782.890360] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.906442] LustreError: Skipped 1 previous similar message [ 7786.464522] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7786.471511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.483266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7786.487996] Lustre: Skipped 3 previous similar messages [ 7788.016261] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7788.035923] LustreError: Skipped 1 previous similar message [ 7792.042198] Lustre: server umount lustre-OST0000 complete [ 7796.192365] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7796.198884] LustreError: Skipped 1 previous similar message [ 7796.200977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.215146] Lustre: Skipped 1 previous similar message [ 7796.227537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.230159] Lustre: Skipped 2 previous similar messages [ 7801.627982] Lustre: server umount lustre-MDT0000 complete [ 7803.373511] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.401433] LustreError: Skipped 1 previous similar message [ 7805.427068] LustreError: 261848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199218 with bad export cookie 4640049135185044862 [ 7805.439801] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7805.444236] LustreError: 261848:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7805.810546] Lustre: server umount lustre-MDT0001 complete [ 7817.932369] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 7820.824431] Key type lgssc unregistered [ 7821.160275] LNet: 267530:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7821.177126] LNet: Removed LNI 192.168.202.115@tcp [ 7821.988595] Key type .llcrypt unregistered [ 7821.991680] Key type ._llcrypt unregistered [ 7860.519551] alg: No test for adler32 (adler32-zlib) [ 7861.274515] Key type ._llcrypt registered [ 7861.277972] Key type .llcrypt registered [ 7861.410474] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7862.858056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7863.395933] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7863.689841] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 7863.698715] LNet: Accept secure, port 988 [ 7865.431132] Key type lgssc registered [ 7866.441594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7875.340795] loop0: detected capacity change from 0 to 1099511627776 [ 7876.075207] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 7876.082075] *** VALIDATE xfs *** [ 7876.101061] XFS (loop0): Mounting V5 Filesystem [ 7876.147650] XFS (loop0): Ending clean mount [ 7877.470297] loop1: detected capacity change from 0 to 18691697672192 [ 7960.396150] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7976.384578] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 7985.089199] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 7993.909593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8000.521984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8006.803639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8012.869836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8020.366666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8020.412569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.647288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8021.700997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8021.769501] Lustre: lustre-MDT0000: new disk, initializing [ 8021.848152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.862757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8026.175649] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8035.785334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8035.854519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.920625] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8035.956381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8035.960366] Lustre: Skipped 1 previous similar message [ 8036.035650] Lustre: lustre-MDT0001: new disk, initializing [ 8036.130157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8036.157466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8036.163639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8041.129527] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8050.033969] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8056.703703] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8064.099338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8064.173943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8064.180529] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8064.524479] Lustre: lustre-OST0000: new disk, initializing [ 8064.527938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8064.637575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.213060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8070.225064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8070.449333] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8080.332785] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8085.025628] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8085.209531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8089.472958] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8089.648846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8096.228396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.233691] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.240308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.250409] Lustre: Skipped 1 previous similar message [ 8097.375200] Lustre: server umount lustre-OST0000 complete [ 8101.346132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.348909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.358951] Lustre: Skipped 2 previous similar messages [ 8101.372295] Lustre: Skipped 1 previous similar message [ 8106.467278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8106.484374] Lustre: Skipped 1 previous similar message [ 8106.633702] Lustre: server umount lustre-MDT0000 complete [ 8109.900357] LustreError: 274568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199522 with bad export cookie 8142354461023326722 [ 8109.917289] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8109.918321] LustreError: 274568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8110.374508] Lustre: server umount lustre-MDT0001 complete [ 8119.566671] XFS (loop0): Unmounting Filesystem [ 8123.718952] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 16:45:36 (1776199536) [ 8130.429412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.054399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.104309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.943069] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8136.161258] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8141.282777] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8142.278791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.496498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.542882] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8151.496766] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8156.249959] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8161.150695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.369480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.912524] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8170.586958] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8183.264280] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8183.265848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.278385] LustreError: Skipped 1 previous similar message [ 8183.279069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.288568] Lustre: Skipped 1 previous similar message [ 8183.303663] Lustre: Skipped 1 previous similar message [ 8188.008544] Lustre: server umount lustre-OST0000 complete [ 8193.504773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.509372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8193.526055] Lustre: Skipped 1 previous similar message [ 8193.539888] Lustre: Skipped 1 previous similar message [ 8198.626158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8198.641755] LustreError: Skipped 1 previous similar message [ 8198.759620] Lustre: server umount lustre-MDT0000 complete [ 8201.373601] LustreError: 277168:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199614 with bad export cookie 8142354461023327576 [ 8201.379148] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8201.386805] LustreError: 277168:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8211.028917] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 8213.546574] Key type lgssc unregistered [ 8213.803626] LNet: 280935:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8213.809485] LNet: Removed LNI 192.168.202.115@tcp [ 8214.573471] Key type .llcrypt unregistered [ 8214.575726] Key type ._llcrypt unregistered [ 8239.353780] alg: No test for adler32 (adler32-zlib) [ 8240.112099] Key type ._llcrypt registered [ 8240.116867] Key type .llcrypt registered [ 8240.218523] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 8241.081647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8241.556659] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 8241.748616] LNet: Added LNI 192.168.202.115@tcp [8/256/0/180] [ 8241.752892] LNet: Accept secure, port 988 [ 8243.439202] Key type lgssc registered [ 8244.343853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8252.046552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.354912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.396192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.360704] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8258.536252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8263.649741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8264.200837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.562886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8269.406307] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8278.106652] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8285.035639] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8290.832191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.312220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.968688] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8303.032020] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8314.216791] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 8314.848289] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8314.857827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.866770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8314.886900] Lustre: Skipped 1 previous similar message [ 8314.906835] Lustre: Skipped 1 previous similar message [ 8319.302231] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 8320.550352] Lustre: server umount lustre-MDT0001 complete [ 8323.049448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8324.422270] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.545883] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.563408] LustreError: Skipped 2 previous similar messages [ 8329.964611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8330.066428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8330.071641] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8330.114255] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8330.210138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8330.212915] Lustre: Skipped 2 previous similar messages [ 8330.412575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8334.689873] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8335.850554] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8335.852537] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8336.371722] LustreError: 286223:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8336.378972] LustreError: 286223:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8336.384427] LustreError: 286223:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8336.396927] LustreError: 282543:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8336.451456] LustreError: 286238:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -17 [ 8336.456552] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 8336.456552] [ 8641.882337] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 8643.055020] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8643.059026] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8652.095898] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 8652.098430] Lustre: Skipped 5 previous similar messages [ 8654.303915] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8654.465102] Lustre: server umount lustre-MDT0001 complete [ 8657.216526] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.774078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8667.062686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.245716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8667.256679] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8667.329495] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8667.915867] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8673.097210] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8673.272024] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8673.278639] Lustre: Skipped 1 previous similar message [ 8673.282448] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8677.390265] LustreError: 295224:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8677.416854] LustreError: 295224:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 8677.425413] LustreError: 295224:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8677.433982] LustreError: 295224:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8677.440616] LustreError: 282543:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8677.446342] LustreError: 282543:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8677.533481] LustreError: 295227:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 8677.549450] LustreError: 295227:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -17 [ 8677.562092] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.115@tcp [ 8677.562092] [ 8978.780083] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.15@tcp (stopping) [ 8978.786525] Lustre: Skipped 3 previous similar messages [ 8980.448483] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.460386] Lustre: Skipped 1 previous similar message [ 8983.749720] Lustre: server umount lustre-MDT0001 complete [ 8983.873803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.886783] LustreError: Skipped 5 previous similar messages [ 8986.855902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8993.795904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.909299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8993.913834] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8993.944672] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8994.255302] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8998.272088] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 8999.411571] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8999.421706] Lustre: Skipped 1 previous similar message [ 8999.435364] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8999.950692] LustreError: 304225:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8999.957524] LustreError: 304225:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8999.962779] LustreError: 304225:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8999.981222] LustreError: 282543:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8999.989856] LustreError: 282543:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9000.086309] LustreError: 304229:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 9000.115466] LustreError: 304229:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.115@tcp: cfg command failed: rc = -114 [ 9000.131476] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 9000.131476] [ 9327.072983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.073805] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9327.076697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9327.076705] Lustre: Skipped 3 previous similar messages [ 9327.079395] Lustre: Skipped 2 previous similar messages [ 9337.311136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9337.462195] Lustre: server umount lustre-MDT0000 complete [ 9340.993417] LustreError: 282531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200754 with bad export cookie 16730354478210220124 [ 9341.000395] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9341.006359] LustreError: 282531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.432446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.432577] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.440685] Lustre: Skipped 1 previous similar message [ 9342.449562] LustreError: Skipped 5 previous similar messages [ 9347.355894] Lustre: server umount lustre-MDT0001 complete [ 9361.118240] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9372.498490] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9382.870950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9389.174658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9395.138829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9401.913963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9408.818436] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 17:07:00 (1776200820) [ 9414.653630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9414.725296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9414.976770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9415.017346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9415.102995] Lustre: lustre-MDT0000: new disk, initializing [ 9415.194606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9415.211459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9419.035973] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 9428.299463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9428.358713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.411621] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9428.435093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9428.436804] Lustre: Skipped 1 previous similar message [ 9428.477265] Lustre: lustre-MDT0001: new disk, initializing [ 9428.521755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9428.538573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9428.545482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9432.380562] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 9442.356173] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9448.445639] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9455.375391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9455.465710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.483565] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9455.845964] Lustre: lustre-OST0000: new disk, initializing [ 9455.850937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9455.996213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.491427] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug -1 all 8 [ 9465.379845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9465.389963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9468.971890] Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9476.065520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.070425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.087533] Lustre: Skipped 1 previous similar message [ 9476.099033] Lustre: Skipped 10 previous similar messages [ 9478.356340] Lustre: server umount lustre-MDT0000 complete [ 9478.358570] Lustre: Skipped 1 previous similar message [ 9481.190463] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.214919] LustreError: Skipped 3 previous similar messages [ 9482.271081] LustreError: 316159:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 9482.281178] LustreError: 315289:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200895 with bad export cookie 16730354478210222035 [ 9482.286642] LustreError: 316159:0:(llog_cat.c:739:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 9482.286658] LustreError: 316159:0:(llog_cat.c:775:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 9482.287223] LustreError: 166-1: MGC192.168.202.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9482.318695] LustreError: 315289:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9492.959302] Lustre: 318543:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776200899/real 1776200899] req@00000000b9158637 x1862480328576832/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776200905 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9527.007612] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_hostid [ 9533.749192] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing load_modules_local [ 9544.061741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9550.854818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9556.727062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9563.002199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9564.369465] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9266 sec ========== 17:09:36 (1776200976) [ 9606.464622] Lustre: DEBUG MARKER: oleg215-server.virtnet: executing unload_modules_local [ 9609.153790] Key type lgssc unregistered [ 9609.390967] LNet: 323653:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9609.402583] LNet: Removed LNI 192.168.202.115@tcp [ 9610.244686] Key type .llcrypt unregistered [ 9610.255435] Key type ._llcrypt unregistered