[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493989463 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002393] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008657] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011017] pid_max: default: 32768 minimum: 301 [ 0.012177] LSM: Security Framework initializing [ 0.014046] Yama: becoming mindful. [ 0.015048] SELinux: Initializing. [ 0.016000] *** VALIDATE selinux *** [ 0.024842] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029586] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031148] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032127] *** VALIDATE tmpfs *** [ 0.034306] *** VALIDATE proc *** [ 0.036023] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040041] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044165] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.047197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048664] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051009] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.056379] rcu: Hierarchical SRCU implementation. [ 0.058474] smp: Bringing up secondary CPUs ... [ 0.059578] x86: Booting SMP configuration: [ 0.060047] .... node #0, CPUs: #1 #2 #3 [ 0.066209] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.172037] node 0 deferred pages initialised in 100ms [ 0.175178] devtmpfs: initialized [ 0.176000] x86/mm: Memory block size: 128MB [ 0.178935] gcov: version magic: 0x41383552 [ 0.182518] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.183134] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.184512] pinctrl core: initialized pinctrl subsystem [ 0.185330] [ 0.185954] ************************************************************* [ 0.186019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187021] ** ** [ 0.188020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189025] ** ** [ 0.190023] ** This means that this kernel is built to expose internal ** [ 0.191027] ** IOMMU data structures, which may compromise security on ** [ 0.192028] ** your system. ** [ 0.193028] ** ** [ 0.194020] ** If you see this message and you are not debugging the ** [ 0.195021] ** kernel, report this immediately to your vendor! ** [ 0.196019] ** ** [ 0.197022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198020] ************************************************************* [ 0.199809] NET: Registered protocol family 16 [ 0.200000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207013] cpuidle: using governor menu [ 0.208779] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211454] PCI: Using configuration type 1 for base access [ 0.213118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227145] cryptd: max_cpu_qlen set to 1000 [ 0.230349] ACPI: Added _OSI(Module Device) [ 0.231015] ACPI: Added _OSI(Processor Device) [ 0.233015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242313] ACPI: Interpreter enabled [ 0.244064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245010] ACPI: Using IOAPIC for interrupt routing [ 0.246148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250571] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260791] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268120] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273400] acpiphp: Slot [2] registered [ 0.274215] acpiphp: Slot [3] registered [ 0.276134] acpiphp: Slot [4] registered [ 0.277118] acpiphp: Slot [5] registered [ 0.278167] acpiphp: Slot [6] registered [ 0.280170] acpiphp: Slot [7] registered [ 0.281180] acpiphp: Slot [8] registered [ 0.283172] acpiphp: Slot [9] registered [ 0.284144] acpiphp: Slot [10] registered [ 0.285105] acpiphp: Slot [11] registered [ 0.287098] acpiphp: Slot [12] registered [ 0.288115] acpiphp: Slot [13] registered [ 0.289115] acpiphp: Slot [14] registered [ 0.292160] acpiphp: Slot [15] registered [ 0.293135] acpiphp: Slot [16] registered [ 0.295109] acpiphp: Slot [17] registered [ 0.296122] acpiphp: Slot [18] registered [ 0.297120] acpiphp: Slot [19] registered [ 0.299092] acpiphp: Slot [20] registered [ 0.300096] acpiphp: Slot [21] registered [ 0.301103] acpiphp: Slot [22] registered [ 0.302113] acpiphp: Slot [23] registered [ 0.304116] acpiphp: Slot [24] registered [ 0.305085] acpiphp: Slot [25] registered [ 0.306087] acpiphp: Slot [26] registered [ 0.308127] acpiphp: Slot [27] registered [ 0.309107] acpiphp: Slot [28] registered [ 0.311099] acpiphp: Slot [29] registered [ 0.312172] acpiphp: Slot [30] registered [ 0.314116] acpiphp: Slot [31] registered [ 0.315082] PCI host bridge to bus 0000:00 [ 0.316019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.326040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333733] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338929] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358036] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364503] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367771] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372608] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393022] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.398018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404255] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435029] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.443789] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.457020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476026] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.486000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.516034] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.525735] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.534030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.540025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.557039] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.571980] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.577023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.584021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.598023] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.608725] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.623027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640029] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.650520] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.652387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.654401] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.657431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.659523] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.664151] iommu: Default domain type: Passthrough [ 0.665000] SCSI subsystem initialized [ 0.665373] ACPI: bus type USB registered [ 0.666000] usbcore: registered new interface driver usbfs [ 0.669122] usbcore: registered new interface driver hub [ 0.671094] usbcore: registered new device driver usb [ 0.673195] pps_core: LinuxPPS API ver. 1 registered [ 0.675013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678090] PTP clock support registered [ 0.683177] EDAC MC: Ver: 3.0.0 [ 0.686210] PCI: Using ACPI for IRQ routing [ 0.687912] NetLabel: Initializing [ 0.689016] NetLabel: domain hash size = 128 [ 0.690010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.692094] NetLabel: unlabeled traffic allowed by default [ 0.694153] vgaarb: loaded [ 0.696374] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.697000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.704341] clocksource: Switched to clocksource kvm-clock [ 0.816491] VFS: Disk quotas dquot_6.6.0 [ 0.818140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.820918] *** VALIDATE ramfs *** [ 0.822354] *** VALIDATE hugetlbfs *** [ 0.823977] pnp: PnP ACPI init [ 0.827396] pnp: PnP ACPI: found 6 devices [ 0.851156] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.854810] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857109] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859764] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.862059] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.864695] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.867285] NET: Registered protocol family 2 [ 0.869994] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.875481] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.879254] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.885390] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.888910] TCP: Hash tables configured (established 65536 bind 65536) [ 0.892952] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.895891] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899096] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902592] NET: Registered protocol family 1 [ 0.906078] RPC: Registered named UNIX socket transport module. [ 0.908412] RPC: Registered udp transport module. [ 0.910207] RPC: Registered tcp transport module. [ 0.912118] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.914623] NET: Registered protocol family 44 [ 0.916775] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.918738] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920782] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.922692] PCI: CLS 0 bytes, default 64 [ 0.924285] Unpacking initramfs... [ 2.368861] debug: unmapping init [mem 0xffff9c2dbcc54000-0xffff9c2dbffbffff] [ 2.374296] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.376672] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.380442] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.870728] Initialise system trusted keyrings [ 2.872579] Key type blacklist registered [ 2.874248] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.885324] zbud: loaded [ 2.889027] *** VALIDATE nfs *** [ 2.890385] *** VALIDATE nfs4 *** [ 2.892036] pstore: using deflate compression [ 2.895599] Platform Keyring initialized [ 3.002756] NET: Registered protocol family 38 [ 3.005830] Key type asymmetric registered [ 3.006938] Asymmetric key parser 'x509' registered [ 3.008723] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.013262] io scheduler mq-deadline registered [ 3.015042] io scheduler kyber registered [ 3.016399] io scheduler bfq registered [ 3.017925] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.020757] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.023741] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.026825] ACPI: Power Button [PWRF] [ 3.118252] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.212939] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.398160] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.490610] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.678794] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.711328] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.741867] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.747409] Non-volatile memory driver v1.3 [ 3.749403] Linux agpgart interface v0.103 [ 3.778788] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.782829] vda: detected capacity change from 0 to 34807808 [ 3.798660] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.801471] vdb: detected capacity change from 0 to 1073741824 [ 3.816737] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.819436] vdc: detected capacity change from 0 to 2621440000 [ 3.833758] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.837044] vdd: detected capacity change from 0 to 2621440000 [ 3.857587] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.860657] vde: detected capacity change from 0 to 4294967296 [ 3.875630] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.877991] vdf: detected capacity change from 0 to 4294967296 [ 3.888312] libphy: Fixed MDIO Bus: probed [ 3.894496] usbcore: registered new interface driver usbserial_generic [ 3.896931] usbserial: USB Serial support registered for generic [ 3.899499] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.903334] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.905347] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.908462] mousedev: PS/2 mouse device common for all mice [ 3.912560] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.912768] rtc_cmos 00:05: RTC can wake from S4 [ 3.919062] rtc_cmos 00:05: registered as rtc0 [ 3.921699] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.921763] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.924351] intel_pstate: CPU model not supported [ 3.930243] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.933971] hid: raw HID events driver (C) Jiri Kosina [ 3.936125] usbcore: registered new interface driver usbhid [ 3.937751] usbhid: USB HID core driver [ 3.939527] drop_monitor: Initializing network drop monitor service [ 3.941676] Initializing XFRM netlink socket [ 3.943602] NET: Registered protocol family 10 [ 3.946394] Segment Routing with IPv6 [ 3.947888] NET: Registered protocol family 17 [ 3.950738] mpls_gso: MPLS GSO support [ 3.955349] RAS: Correctable Errors collector initialized. [ 3.957703] AVX version of gcm_enc/dec engaged. [ 3.959021] AES CTR mode by8 optimization enabled [ 4.050755] sched_clock: Marking stable (4050670308, 0)->(4996470539, -945800231) [ 4.055321] registered taskstats version 1 [ 4.057172] Loading compiled-in X.509 certificates [ 4.058881] zswap: loaded using pool lzo/zbud [ 4.087611] Key type big_key registered [ 4.101530] Key type encrypted registered [ 4.103164] ima: No TPM chip found, activating TPM-bypass! [ 4.106124] ima: Allocated hash algorithm: sha1 [ 4.108229] ima: No architecture policies found [ 4.110154] evm: Initialising EVM extended attributes: [ 4.112466] evm: security.selinux [ 4.113814] evm: security.ima [ 4.115401] evm: security.capability [ 4.116859] evm: HMAC attrs: 0x1 [ 4.120328] rtc_cmos 00:05: setting system clock to 2025-10-24 13:54:39 UTC (1761314079) [ 4.127668] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 4.130588] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 4.140069] Write protecting the kernel read-only data: 28672k [ 4.143100] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 4.146351] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 4.191615] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.202126] systemd[1]: Detected virtualization kvm. [ 4.204411] systemd[1]: Detected architecture x86-64. [ 4.206358] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.233048] systemd[1]: No hostname configured. [ 4.234744] systemd[1]: Set hostname to . [ 4.236626] random: systemd: uninitialized urandom read (16 bytes read) [ 4.238898] systemd[1]: Initializing machine ID from random generator. [ 4.286535] random: ln: uninitialized urandom read (6 bytes read) [ 4.392869] random: systemd: uninitialized urandom read (16 bytes read) [ 4.395484] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.399556] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.404226] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.056171] device-mapper: uevent: version 1.0.3 [ 5.058584] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 5.627058] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.815332] virtio_net virtio0 ens2: renamed from eth0 [ 5.988066] scsi host0: ata_piix [ 6.009575] scsi host1: ata_piix [ 6.011209] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.013815] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.708976] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.624114] random: crng init done [ 10.625449] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.126716] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.382693] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.673926] SELinux: Disabled at runtime. [ 12.734305] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.743470] systemd[1]: Detected virtualization kvm. [ 12.745416] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.339858] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.343516] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.349605] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.353580] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.356754] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.366268] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.371868] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ 13.466081] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.937382] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.288898] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.314048] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.464323] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.481825] EDAC sbridge: Ver: 1.1.2 [ 16.408065] Key type dns_resolver registered [ 16.727967] NFS: Registering the id_resolver key type [ 16.729942] Key type id_resolver registered [ 16.731553] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg214-server login: [ 28.601141] hrtimer: interrupt took 3127859 ns [ 61.207447] libcfs: loading out-of-tree module taints kernel. [ 61.254278] alg: No test for adler32 (adler32-zlib) [ 62.012240] Key type ._llcrypt registered [ 62.013653] Key type .llcrypt registered [ 62.159542] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 79.923427] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 81.526854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.258069] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 83.180941] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 83.184726] LNet: Accept secure, port 988 [ 85.048184] Key type lgssc registered [ 86.954904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.683327] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 108.491700] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.540371] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.565136] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.423594] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.325479] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 158.616710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 158.683659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.977359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 160.013106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 160.159597] Lustre: lustre-MDT0000: new disk, initializing [ 160.295565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.330578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.843415] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 179.727608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.863498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.018764] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 180.067341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 180.074450] Lustre: Skipped 1 previous similar message [ 180.194540] Lustre: lustre-MDT0001: new disk, initializing [ 180.251231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.277205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.281516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.877529] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 200.066268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.157953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.161233] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 200.308415] Lustre: lustre-OST0000: new disk, initializing [ 200.318421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 200.375667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.545657] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 207.917488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.923523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.210563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.326777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.334688] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 218.422189] Lustre: lustre-OST0001: new disk, initializing [ 218.430201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 218.500702] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.749343] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 227.375327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.393792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.442719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.775064] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 248.245522] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.520902] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing yml_node [ 256.954551] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 259.182498] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 261.423886] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 262.845201] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:58:56 EDT 2025 [ 270.438467] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 271.867933] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 287.714885] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.724587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.747874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.744379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.744594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.753303] Lustre: Skipped 2 previous similar messages [ 288.757353] Lustre: Skipped 2 previous similar messages [ 292.736254] Lustre: server umount lustre-MDT0000 complete [ 293.857675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.871250] LustreError: Skipped 3 previous similar messages [ 295.943715] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314371 with bad export cookie 12511957948943985181 [ 295.948445] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.976818] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.981822] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.981892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.981896] Lustre: Skipped 1 previous similar message [ 299.000687] Lustre: Skipped 1 previous similar message [ 299.017307] LustreError: Skipped 1 previous similar message [ 301.024780] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.031850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.044401] LustreError: Skipped 1 previous similar message [ 301.058687] Lustre: Skipped 2 previous similar messages [ 307.172798] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.176188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.183675] LustreError: Skipped 3 previous similar messages [ 307.188973] Lustre: Skipped 3 previous similar messages [ 310.240265] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 310.500660] Lustre: server umount lustre-MDT0001 complete [ 313.924698] Lustre: server umount lustre-OST0000 complete [ 317.284439] Lustre: server umount lustre-OST0001 complete [ 323.277988] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 331.069407] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 340.440253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.545101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.825100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.372468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.360733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.451594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.736409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 364.771843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 364.905846] Lustre: lustre-MDT0000: new disk, initializing [ 365.029459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.045909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.907785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 378.763332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 378.835389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.900987] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 378.948912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 378.952858] Lustre: Skipped 1 previous similar message [ 379.040886] Lustre: lustre-MDT0001: new disk, initializing [ 379.131057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.194940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.225935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 383.222542] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 392.854259] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 398.975539] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 405.710279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 405.823406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.843618] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 406.210765] Lustre: lustre-OST0000: new disk, initializing [ 406.217545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 406.300965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.868483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.886219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 410.152779] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 418.394648] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 422.371692] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 422.629930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 426.610465] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 426.797647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.637791] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.640531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.641121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.663839] Lustre: Skipped 1 previous similar message [ 434.890840] Lustre: server umount lustre-OST0000 complete [ 440.801509] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.809647] LustreError: Skipped 1 previous similar message [ 440.813747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.194483] Lustre: server umount lustre-MDT0000 complete [ 448.993619] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.030351] LustreError: Skipped 1 previous similar message [ 449.227659] LustreError: 14339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314524 with bad export cookie 12511957948943986637 [ 449.230704] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.247713] LustreError: 14339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 455.094895] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:02:07 (1761314527) [ 462.920945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.336567] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.396910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.482776] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 476.634860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.805753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 488.939786] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 497.436175] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.950984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.944908] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 517.693415] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 521.319851] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 528.356852] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.360449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.366979] Lustre: Skipped 1 previous similar message [ 528.370799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.373675] Lustre: Skipped 4 previous similar messages [ 531.907325] Lustre: server umount lustre-MDT0000 complete [ 531.908905] Lustre: Skipped 1 previous similar message [ 533.990813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.001369] LustreError: Skipped 4 previous similar messages [ 534.897776] LustreError: 18742:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314610 with bad export cookie 12511957948943987491 [ 534.899673] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.907673] LustreError: 18742:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 543.715192] Lustre: 21868:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314613/real 1761314613] req@0000000096e11705 x1846871754621632/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314619 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 561.139781] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 570.539619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.978066] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.077569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.082530] Lustre: Skipped 2 previous similar messages [ 575.194563] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 582.942444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.713433] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 595.615358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.097154] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 608.434978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 608.519242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.531119] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 608.693852] Lustre: lustre-OST0001: new disk, initializing [ 608.701584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 610.343775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 610.353282] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 613.484603] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 622.948539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 626.333263] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 641.506743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.512908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.542353] Lustre: Skipped 4 previous similar messages [ 641.567467] Lustre: Skipped 5 previous similar messages [ 646.727591] Lustre: server umount lustre-MDT0000 complete [ 646.741626] Lustre: Skipped 2 previous similar messages [ 650.611962] LustreError: 23233:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314725 with bad export cookie 12511957948943988338 [ 650.620971] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.634868] LustreError: 23233:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 658.848166] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314727/real 1761314727] req@00000000f80063e2 x1846871754652416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314734 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 660.960165] Lustre: 27064:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314730/real 1761314730] req@0000000098a66354 x1846871754652544/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314736 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 663.013345] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314731/real 1761314731] req@00000000f054daf0 x1846871754652608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314738 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 678.126359] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 685.558644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.024792] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.043404] LustreError: Skipped 3 previous similar messages [ 690.532883] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 699.136479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.591050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 699.597905] Lustre: Skipped 4 previous similar messages [ 703.634137] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 712.221989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.793607] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 724.662570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 729.039130] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 736.786290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 740.203883] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 740.206519] Lustre: Skipped 1 previous similar message [ 750.569696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.573100] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 750.578202] Lustre: Skipped 3 previous similar messages [ 750.580274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.596129] Lustre: Skipped 7 previous similar messages [ 756.777223] Lustre: server umount lustre-MDT0000 complete [ 756.779906] Lustre: Skipped 3 previous similar messages [ 760.228413] LustreError: 28477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314835 with bad export cookie 12511957948943989584 [ 760.229796] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.236806] LustreError: 28477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 767.840183] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314836/real 1761314836] req@00000000a99890cf x1846871754677440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314843 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 767.860956] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 781.645983] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:07:34 (1761314854) [ 805.626119] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 815.082642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.621752] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.644630] LustreError: Skipped 7 previous similar messages [ 820.981316] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 829.146158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.916760] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 842.096897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.732316] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 855.069151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.263947] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 867.231420] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 870.803922] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 880.621147] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 880.626048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.640676] Lustre: Skipped 3 previous similar messages [ 880.647301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 880.653329] Lustre: Skipped 4 previous similar messages [ 886.663050] Lustre: server umount lustre-MDT0000 complete [ 886.665120] Lustre: Skipped 3 previous similar messages [ 890.260040] LustreError: 34321:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314965 with bad export cookie 12511957948943990655 [ 890.266572] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.272496] LustreError: 34321:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 898.464190] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314966/real 1761314966] req@0000000008fcb45d x1846871754701760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314973 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 898.501184] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 911.498752] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:09:44 (1761314984) [ 925.148648] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 934.796526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.486926] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 948.600723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.673816] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 963.295100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.827307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.840255] Lustre: Skipped 8 previous similar messages [ 968.087978] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 978.261512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.658388] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 990.202246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 999.678678] Lustre: Setting parameter general.mdd.*.changelog_mask in log params [ 999.687475] Lustre: Skipped 1 previous similar message [ 1012.519760] LustreError: 39502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315087 with bad export cookie 12511957948943991726 [ 1012.526956] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.549302] LustreError: 39502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1017.313530] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.325574] Lustre: Skipped 9 previous similar messages [ 1027.707516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.607820] Lustre: Evicted from MGS (at 192.168.202.114@tcp) after server handle changed from 0xada367769f6e83ae to 0xada367769f6e880e [ 1029.647223] Lustre: MGC192.168.202.114@tcp: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 1030.081898] BUG: spinlock bad magic on CPU#3, lctl/43433 [ 1030.094103] lock: 0xffff9c2e3f621d00, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1030.100861] CPU: 3 PID: 43433 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1030.109334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1030.118337] Call Trace: [ 1030.123841] ? dump_stack+0xbb/0x10e [ 1030.125830] ? spin_dump.cold.3+0x32/0x65 [ 1030.127181] ? do_raw_spin_lock+0xce/0x190 [ 1030.130098] ? _raw_spin_lock+0x21/0x30 [ 1030.132206] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 1030.137993] ? __seccomp_filter+0x4b/0x680 [ 1030.142317] ? full_proxy_write+0x5e/0xa0 [ 1030.143476] ? __vfs_write+0x1c/0x60 [ 1030.148153] ? vfs_write+0xd8/0x2b0 [ 1030.150208] ? ksys_write+0x66/0x120 [ 1030.153674] ? __x64_sys_write+0x1e/0x30 [ 1030.160547] ? do_syscall_64+0xc1/0x440 [ 1030.161901] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1034.776438] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1044.178964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.658431] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1048.873753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1056.391215] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.129769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1063.701259] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1065.230778] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1070.049663] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.752995] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.775890] LustreError: Skipped 34 previous similar messages [ 1079.617650] LustreError: 39502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315154 with bad export cookie 12511957948943992846 [ 1079.622225] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.628629] LustreError: 39502:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1090.528192] Lustre: 45655:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315159/real 1761315159] req@00000000fbed9088 x1846871754748096/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315165 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1090.540816] Lustre: 45655:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1102.625643] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:12:55 (1761315175) [ 1118.162992] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1127.353880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.139627] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1142.728583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.091946] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1157.692898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1162.638704] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1171.537587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1176.019187] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1183.273781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1204.707926] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.712941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1204.723609] Lustre: Skipped 28 previous similar messages [ 1206.315405] Lustre: server umount lustre-MDT0000 complete [ 1206.317397] Lustre: Skipped 9 previous similar messages [ 1210.964358] LustreError: 47264:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315286 with bad export cookie 12511957948943993658 [ 1210.969924] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.988432] LustreError: 47264:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1220.068434] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315288/real 1761315288] req@00000000cd0f56e7 x1846871754776832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315295 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1220.097230] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1233.844461] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:15:06 (1761315306) [ 1235.901284] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1237.782548] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:15:10 (1761315310) [ 1249.822810] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 1259.217594] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1269.138344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1276.706367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1284.057287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1284.135269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.350322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1284.353837] Lustre: Skipped 2 previous similar messages [ 1284.403928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1284.535909] Lustre: lustre-MDT0000: new disk, initializing [ 1284.660697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1289.572604] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1299.431498] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.602652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1305.676608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.682346] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1305.931879] Lustre: lustre-OST0000: new disk, initializing [ 1305.934474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1305.936840] Lustre: Skipped 1 previous similar message [ 1310.252418] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1319.261202] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1321.317319] Lustre: DEBUG MARKER: create test files [ 1343.970466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.978587] Lustre: Skipped 10 previous similar messages [ 1358.816248] Lustre: 55575:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315428/real 1761315428] req@0000000022b87566 x1846871754785984/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315434 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1358.844145] Lustre: 55575:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 1369.824866] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1372.293920] Key type lgssc unregistered [ 1372.502059] LNet: 56179:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.509533] LNet: Removed LNI 192.168.202.114@tcp [ 1373.170507] Key type .llcrypt unregistered [ 1373.172517] Key type ._llcrypt unregistered [ 1401.750867] alg: No test for adler32 (adler32-zlib) [ 1402.524482] Key type ._llcrypt registered [ 1402.526069] Key type .llcrypt registered [ 1402.666915] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1404.561036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.005180] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 1405.202352] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1405.210052] LNet: Accept secure, port 988 [ 1406.928146] Key type lgssc registered [ 1407.980734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.050791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.594466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.905823] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1426.383899] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1433.044745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.424440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.409212] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1444.958413] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1446.713241] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1446.807810] Lustre: 59043:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1446.816089] Lustre: 59043:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1446.822882] Lustre: 59043:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1446.828819] Lustre: 59043:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1446.832248] Lustre: 59043:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1446.836285] Lustre: 59043:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1456.347314] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1460.968263] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1467.361604] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.364533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.386378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.940159] Lustre: server umount lustre-OST0000 complete [ 1478.624922] Lustre: 60075:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315548/real 1761315548] req@0000000010c88769 x1846873140763200/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315554 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1478.821624] Lustre: server umount lustre-MDT0000 complete [ 1489.742527] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1492.749411] Key type lgssc unregistered [ 1493.066703] LNet: 60679:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.080909] LNet: Removed LNI 192.168.202.114@tcp [ 1493.838610] Key type .llcrypt unregistered [ 1493.851595] Key type ._llcrypt unregistered [ 1521.949773] alg: No test for adler32 (adler32-zlib) [ 1522.708466] Key type ._llcrypt registered [ 1522.709641] Key type .llcrypt registered [ 1522.837781] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1523.982151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1524.399501] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 1524.609246] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1524.612339] LNet: Accept secure, port 988 [ 1526.272165] Key type lgssc registered [ 1527.219316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.080300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.595037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.477693] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1546.859723] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1553.860532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.285339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.662531] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1565.515552] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1566.579523] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1566.722876] Lustre: 63617:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1566.734272] Lustre: 63617:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1566.745238] Lustre: 63617:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1566.752277] Lustre: 63617:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1566.757149] Lustre: 63617:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1566.763663] Lustre: 63617:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1575.546384] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1587.172455] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.176432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.205407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1588.476348] Lustre: server umount lustre-OST0000 complete [ 1596.897545] Lustre: 64582:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315666/real 1761315666] req@00000000e66ed2af x1846873265543616/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315672 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1597.061449] Lustre: server umount lustre-MDT0000 complete [ 1607.272940] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 1609.658735] Key type lgssc unregistered [ 1609.949113] LNet: 65186:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.958836] LNet: Removed LNI 192.168.202.114@tcp [ 1610.772686] Key type .llcrypt unregistered [ 1610.777122] Key type ._llcrypt unregistered [ 1626.183640] alg: No test for adler32 (adler32-zlib) [ 1626.940438] Key type ._llcrypt registered [ 1626.947122] Key type .llcrypt registered [ 1627.051135] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 1639.140655] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1640.235342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.656420] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 1640.861726] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 1640.865745] LNet: Accept secure, port 988 [ 1642.512143] Key type lgssc registered [ 1643.473694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.505739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1656.230448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1661.800752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.345989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1672.917366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1673.002040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.298953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1674.323812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1674.387611] Lustre: lustre-MDT0000: new disk, initializing [ 1674.471894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.497291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1678.361542] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1687.260909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1687.327903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.395463] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1687.467213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1687.470695] Lustre: Skipped 1 previous similar message [ 1687.568488] Lustre: lustre-MDT0001: new disk, initializing [ 1687.676847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.726214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1687.738370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1691.104844] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1700.231576] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1704.508244] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1711.739489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1711.832853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.838577] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1712.127910] Lustre: lustre-OST0000: new disk, initializing [ 1712.136750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1712.246056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.208361] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1717.824649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1717.836530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1726.024623] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.604464] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1730.828978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1735.073658] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1735.336300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1738.721037] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1738.726037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.735106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.764685] Lustre: Skipped 1 previous similar message [ 1743.291383] Lustre: server umount lustre-OST0000 complete [ 1748.964211] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1748.967438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.974507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.981392] Lustre: Skipped 2 previous similar messages [ 1754.085230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.088165] Lustre: Skipped 1 previous similar message [ 1760.227974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1760.230874] Lustre: Skipped 3 previous similar messages [ 1761.248152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1761.385045] Lustre: server umount lustre-MDT0000 complete [ 1765.005442] LustreError: 68457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315840 with bad export cookie 4981779173768205560 [ 1765.013071] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.375077] Lustre: server umount lustre-MDT0001 complete [ 1773.638902] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:24:06 (1761315846) [ 1781.661240] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1794.592361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.318969] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 1825.905762] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 1835.274719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1841.915797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.089688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1853.190153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.146644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1860.237979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.476064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1860.507835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1860.617916] Lustre: lustre-MDT0000: new disk, initializing [ 1860.707739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.740594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1864.975802] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1874.284510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1874.368358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.466711] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1874.537793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1874.551413] Lustre: Skipped 1 previous similar message [ 1874.697748] Lustre: lustre-MDT0001: new disk, initializing [ 1874.813310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.864685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1874.868916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1878.713159] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1887.507622] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1892.712129] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1898.845701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1898.960948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.970396] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1899.198482] Lustre: lustre-OST0000: new disk, initializing [ 1899.204340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1899.306261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.533466] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1907.772278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1907.776680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1912.389439] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1917.065753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1917.274545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1921.608428] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1921.854937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1928.679119] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1928.679176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.682564] LustreError: Skipped 1 previous similar message [ 1928.683374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.687639] Lustre: Skipped 2 previous similar messages [ 1930.104816] Lustre: server umount lustre-OST0000 complete [ 1933.794816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.804856] Lustre: Skipped 1 previous similar message [ 1938.920052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.922427] Lustre: Skipped 3 previous similar messages [ 1948.129070] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1948.272863] Lustre: server umount lustre-MDT0000 complete [ 1950.188475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.227600] LustreError: Skipped 1 previous similar message [ 1952.284019] LustreError: 76751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316027 with bad export cookie 4981779173768206414 [ 1952.288502] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.290662] LustreError: 76751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1960.432250] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:27:13 (1761316033) [ 1966.404865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.847670] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.918539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.688516] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1972.195023] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.317054] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.162292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.205874] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 1991.067751] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.352042] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.169910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.518092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.521404] Lustre: Skipped 1 previous similar message [ 2008.263639] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2014.138442] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.370594] Lustre: 84059:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 943f2405-2e19-4c7d-9513-a8c3664c9d56 at adminstrative request [ 2021.864545] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.879486] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2028.017356] Lustre: MGS: Client 943f2405-2e19-4c7d-9513-a8c3664c9d56 (at 0@lo) reconnecting [ 2028.029560] Lustre: Evicted from MGS (at 192.168.202.114@tcp) after server handle changed from 0x4522d5c391e49ba4 to 0x4522d5c391e49e44 [ 2028.054238] Lustre: MGC192.168.202.114@tcp: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 2056.436525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2056.525076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.528762] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2056.655846] Lustre: lustre-OST0001: new disk, initializing [ 2056.659270] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2062.282610] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2065.003484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2065.015336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2073.300592] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2080.741856] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2080.747513] LustreError: Skipped 1 previous similar message [ 2080.750919] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.755778] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2080.762631] Lustre: Skipped 1 previous similar message [ 2080.785234] Lustre: Skipped 5 previous similar messages [ 2082.187703] Lustre: server umount lustre-OST0001 complete [ 2082.189674] Lustre: Skipped 1 previous similar message [ 2085.862822] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.876215] LustreError: Skipped 1 previous similar message [ 2096.100259] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.110071] LustreError: Skipped 3 previous similar messages [ 2100.192487] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2100.399812] Lustre: server umount lustre-OST0000 complete [ 2106.337792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.347444] Lustre: Skipped 3 previous similar messages [ 2114.780849] LustreError: 84454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316190 with bad export cookie 4981779173768207940 [ 2114.785304] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.793628] LustreError: 84454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2123.083703] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:29:56 (1761316196) [ 2124.600551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2126.355331] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:29:59 (1761316199) [ 2127.792757] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2129.280652] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:30:02 (1761316202) [ 2130.867728] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2132.924759] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:30:05 (1761316205) [ 2134.635226] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2139.195251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2140.148819] loop: module loaded [ 2140.165108] loop0: detected capacity change from 0 to 4294967296 [ 2140.244198] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2141.986675] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2145.445473] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2148.943969] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2156.783050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2160.793865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2164.310392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2168.540273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2174.991112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2175.048663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2175.052333] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2175.059211] LustreError: 88512:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2175.063135] LustreError: 88512:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2175.066669] LustreError: 88512:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2175.071836] LustreError: 88512:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2190.833792] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 2200.182471] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2209.010279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2215.748777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2222.167920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2228.481057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2235.209317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2235.315813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.738265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2235.777970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2235.910443] Lustre: lustre-MDT0000: new disk, initializing [ 2235.994799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.997306] Lustre: Skipped 1 previous similar message [ 2236.026482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2241.068547] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2252.610410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2252.723069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.873311] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2253.171378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2257.534625] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2267.842588] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2273.816981] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.398534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.512742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.521852] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2284.445635] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2290.757435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2293.328343] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.086740] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2297.324718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2300.939637] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2301.120408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2306.529915] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.533247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.548392] LustreError: Skipped 1 previous similar message [ 2306.551406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.561473] Lustre: Skipped 1 previous similar message [ 2306.563627] Lustre: Skipped 8 previous similar messages [ 2317.280209] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2317.557686] Lustre: server umount lustre-OST0000 complete [ 2317.560382] Lustre: Skipped 2 previous similar messages [ 2332.038418] LustreError: 93456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316407 with bad export cookie 4981779173768208521 [ 2332.042329] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.065110] LustreError: 93456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.078534] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:33:34 (1761316414) [ 2350.006928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.718265] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.738544] LustreError: Skipped 2 previous similar messages [ 2355.186401] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2363.387523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.976411] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2376.386265] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2382.355887] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.811608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2389.157238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.162396] Lustre: Skipped 4 previous similar messages [ 2393.312572] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2400.127562] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2405.920777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2405.986475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2405.993424] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2406.131306] Lustre: lustre-OST0001: new disk, initializing [ 2406.133617] Lustre: Skipped 2 previous similar messages [ 2406.139600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2406.142158] Lustre: Skipped 3 previous similar messages [ 2409.467609] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2413.662441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2413.672481] Lustre: Skipped 2 previous similar messages [ 2413.683737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2418.200683] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2434.983570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2457.443834] Lustre: Failing over lustre-MDT0000 [ 2458.006345] Lustre: server umount lustre-MDT0000 complete [ 2458.015425] Lustre: Skipped 2 previous similar messages [ 2460.132706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.137054] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.150438] Lustre: Skipped 4 previous similar messages [ 2460.169132] LustreError: Skipped 5 previous similar messages [ 2467.296204] Lustre: 66685:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316535/real 1761316535] req@00000000ea7d82d5 x1846873388346048/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316542 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2467.313645] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2481.521102] LDISKFS-fs (dm-0): recovery complete [ 2481.525077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.689149] Lustre: Evicted from MGS (at 192.168.202.114@tcp) after server handle changed from 0x4522d5c391e4a3df to 0x4522d5c391e531c2 [ 2483.706453] Lustre: MGC192.168.202.114@tcp: Connection restored to (at 0@lo) [ 2483.947325] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2484.578661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2484.581191] Lustre: 99511:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2487.502161] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2489.317776] Lustre: 95926:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2489.323381] Lustre: 95926:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2489.335291] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 2489.384277] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2489.712211] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2490.034932] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2490.040314] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2490.380308] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2490.382618] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2490.389588] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2490.397371] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2491.351048] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2491.362464] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2491.696151] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2491.706256] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2492.700249] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2492.729624] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 2493.370977] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2493.398119] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2493.731153] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2493.746187] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [ 2496.921265] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2496.929395] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2497.570463] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2497.585504] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2497.904200] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2497.911875] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2505.225222] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2505.234511] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 2505.884030] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2505.896911] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2506.216198] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2506.221140] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2521.435414] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2521.453809] Lustre: 101107:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2522.096585] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2522.102988] LustreError: 101107:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2522.424981] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2522.428946] LustreError: 101107:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2544.216471] Lustre: 101107:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2544.228166] Lustre: 101107:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2544.237128] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2544.240075] LustreError: 101107:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2544.246753] Lustre: 101107:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@000000003b372afc x1846873375665600/t0(8589934769) o101->31b5bff9-4388-424f-93a1-13aac3af56ac@192.168.202.14@tcp:530/0 lens 592/0 e 0 to 0 dl 1761316625 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2544.261539] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2544.266488] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2544.276345] Lustre: Skipped 2 previous similar messages [ 2544.292756] Lustre: lustre-OST0001: deleting orphan objects from 0x0:45 to 0x0:545 [ 2544.294192] Lustre: lustre-OST0000: deleting orphan objects from 0x0:77 to 0x0:577 [ 2553.316481] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.320650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.328048] Lustre: Skipped 1 previous similar message [ 2553.336645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2553.339636] Lustre: Skipped 9 previous similar messages [ 2566.112787] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2570.209143] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.212141] LustreError: Skipped 1 previous similar message [ 2571.236925] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.256611] LustreError: Skipped 37 previous similar messages [ 2588.669789] LustreError: 95907:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316664 with bad export cookie 4981779173768245698 [ 2588.678171] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.023500] Lustre: server umount lustre-MDT0001 complete [ 2589.027341] Lustre: Skipped 3 previous similar messages [ 2596.791958] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:37:49 (1761316669) [ 2603.289386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2603.300793] Lustre: *** cfs_fail_loc=197, val=0*** [ 2620.896599] LustreError: 103133:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 1: rc = -5 [ 2631.139593] LustreError: 103133:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [ 2637.312842] Lustre: *** cfs_fail_loc=197, val=0*** [ 2637.314419] Lustre: Skipped 65 previous similar messages [ 2660.960724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.963906] Lustre: Skipped 2 previous similar messages [ 2665.302046] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2672.116938] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2680.904714] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:39:14 (1761316754) [ 2699.871312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.298657] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.307735] LustreError: Skipped 4 previous similar messages [ 2703.408758] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2710.744864] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2713.928102] LustreError: 104789:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2713.937051] LustreError: 104789:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2713.945299] LustreError: 104789:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2720.224198] Lustre: 105480:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316789/real 1761316789] req@000000003368a2c7 x1846873388379264/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316795 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2732.991687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2747.741747] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 2755.964869] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2766.388826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.666394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2779.486584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2785.834854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.187834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.287289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.486662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2793.514051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2793.593144] Lustre: lustre-MDT0000: new disk, initializing [ 2793.721808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2798.302431] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2808.315473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.385659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.668246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2812.978622] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2822.330075] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.916148] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2837.278379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2837.338993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.348185] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2841.810433] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 2850.855587] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2854.410534] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2854.562091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2857.915976] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2858.114563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2860.513030] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.521203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.525101] LustreError: Skipped 2 previous similar messages [ 2860.526847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.526854] Lustre: Skipped 13 previous similar messages [ 2860.563352] Lustre: Skipped 6 previous similar messages [ 2874.336170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2874.470583] Lustre: server umount lustre-OST0000 complete [ 2874.472237] Lustre: Skipped 2 previous similar messages [ 2888.036439] LustreError: 111199:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316963 with bad export cookie 4981779173768256254 [ 2888.039049] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2888.062863] LustreError: 111199:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2899.094506] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:42:51 (1761316971) [ 2910.601469] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 2914.284623] Key type lgssc unregistered [ 2914.732540] LNet: 113793:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.751362] LNet: Removed LNI 192.168.202.114@tcp [ 2915.836823] Key type .llcrypt unregistered [ 2915.839091] Key type ._llcrypt unregistered [ 2935.085157] alg: No test for adler32 (adler32-zlib) [ 2935.837412] Key type ._llcrypt registered [ 2935.839068] Key type .llcrypt registered [ 2935.939626] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 2949.534182] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 2950.255874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2950.813893] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 2950.995817] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 2951.002496] LNet: Accept secure, port 988 [ 2952.675220] Key type lgssc registered [ 2953.788735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2963.332640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2970.524884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2977.116436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2983.181816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2988.049292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.684913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.764740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.009428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2995.031916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2995.094297] Lustre: lustre-MDT0000: new disk, initializing [ 2995.181642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.202148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2998.962495] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3007.736822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3007.819976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.826187] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3007.996934] Lustre: lustre-OST0000: new disk, initializing [ 3008.007422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3008.018159] Lustre: Skipped 1 previous similar message [ 3008.079874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.844399] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3020.260768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.326725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.332473] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3020.468895] Lustre: lustre-OST0001: new disk, initializing [ 3020.475465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3020.544601] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.201576] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3034.084535] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.086451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.108475] Lustre: Skipped 1 previous similar message [ 3037.819805] Lustre: server umount lustre-MDT0000 complete [ 3043.241202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3046.371448] Lustre: 115315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317114/real 1761317114] req@000000003666b13f x1846874761866048/t0(0) o400->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317121 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3046.389461] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.157764] Lustre: server umount lustre-OST0000 complete [ 3050.506272] Lustre: server umount lustre-OST0001 complete [ 3057.977611] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 10:45:31 (1761317131) [ 3064.056786] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3070.987913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.712413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3075.449225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.170069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3080.930892] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:45:54 (1761317154) [ 3090.752285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3092.261739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3093.995877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.874212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3096.859222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3098.496108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.321587] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 3121.077718] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3130.779659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.095391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3142.204618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.423121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3153.554734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3153.614185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.808652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3153.836529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3153.893395] Lustre: lustre-MDT0000: new disk, initializing [ 3153.964593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.985019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3157.864367] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3166.623609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3166.672810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.757979] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3166.796161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3166.798709] Lustre: Skipped 1 previous similar message [ 3166.844604] Lustre: lustre-MDT0001: new disk, initializing [ 3166.928482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3166.960902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3166.975243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3170.808841] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3178.870410] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3183.759226] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3189.034289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.090480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.096475] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3189.265839] Lustre: lustre-OST0000: new disk, initializing [ 3189.273343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3189.351553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.097379] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3195.949992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3195.958359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3200.697075] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3204.467926] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3204.668967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3208.639303] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3208.842598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3211.235815] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.243686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.252801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.255527] Lustre: Skipped 1 previous similar message [ 3211.752243] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3216.760365] Lustre: server umount lustre-OST0000 complete [ 3221.985704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.996582] Lustre: Skipped 1 previous similar message [ 3222.009444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.014506] Lustre: Skipped 2 previous similar messages [ 3227.105603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.110962] Lustre: Skipped 1 previous similar message [ 3233.249482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3233.251970] Lustre: Skipped 3 previous similar messages [ 3234.272793] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3234.353311] Lustre: server umount lustre-MDT0000 complete [ 3237.239789] LustreError: 125838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317312 with bad export cookie 3599374159648057375 [ 3237.260536] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.178138] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:48:38 (1761317318) [ 3250.471822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.764498] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.809381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.665678] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3256.289800] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.885912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.091470] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3275.283937] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3282.780429] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3289.569947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.150132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.160361] Lustre: Skipped 1 previous similar message [ 3294.460242] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3301.908538] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3305.864309] Lustre: *** cfs_fail_loc=159, val=0*** [ 3305.866480] Lustre: Skipped 1 previous similar message [ 3313.058552] Lustre: lustre-MDT0000: Client 98516b08-a4f0-4abc-8eff-26753709ec9e (at 192.168.202.14@tcp) reconnecting [ 3314.061543] Lustre: *** cfs_fail_loc=159, val=0*** [ 3314.064951] Lustre: Skipped 5 previous similar messages [ 3321.217722] Lustre: lustre-MDT0000: Client 98516b08-a4f0-4abc-8eff-26753709ec9e (at 192.168.202.14@tcp) reconnecting [ 3326.947811] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3326.950519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.960869] LustreError: Skipped 1 previous similar message [ 3326.972059] Lustre: Skipped 1 previous similar message [ 3326.984493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.009670] Lustre: Skipped 1 previous similar message [ 3330.281427] Lustre: server umount lustre-OST0000 complete [ 3330.283503] Lustre: Skipped 1 previous similar message [ 3337.185795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.193774] Lustre: Skipped 1 previous similar message [ 3342.312015] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.343561] LustreError: Skipped 1 previous similar message [ 3343.991829] LustreError: 130375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317419 with bad export cookie 3599374159648058229 [ 3343.996904] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3344.017255] LustreError: 130375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3356.449414] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3359.011461] Key type lgssc unregistered [ 3359.300441] LNet: 134252:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.312309] LNet: Removed LNI 192.168.202.114@tcp [ 3360.064176] Key type .llcrypt unregistered [ 3360.070350] Key type ._llcrypt unregistered [ 3382.715474] alg: No test for adler32 (adler32-zlib) [ 3383.482970] Key type ._llcrypt registered [ 3383.489743] Key type .llcrypt registered [ 3383.686143] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3385.254592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3385.625939] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 3385.819147] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3385.821770] LNet: Accept secure, port 988 [ 3387.520155] Key type lgssc registered [ 3388.391128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3394.539862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.790676] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.829689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.571399] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3401.200815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.973420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.218892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3407.127125] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3412.957831] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3417.010454] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3421.841507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.168089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.241753] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3426.302428] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3430.846127] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3442.667294] Lustre: *** cfs_fail_loc=159, val=0*** [ 3450.273853] Lustre: lustre-MDT0000: Client 2d38f3eb-be63-41fc-b042-017c3500aeba (at 192.168.202.14@tcp) reconnecting [ 3452.134570] Lustre: *** cfs_fail_loc=159, val=0*** [ 3459.495497] Lustre: lustre-MDT0001: Client 2d38f3eb-be63-41fc-b042-017c3500aeba (at 192.168.202.14@tcp) reconnecting [ 3460.553137] Lustre: *** cfs_fail_loc=159, val=0*** [ 3460.554865] Lustre: Skipped 3 previous similar messages [ 3467.658067] Lustre: lustre-MDT0001: Client 2d38f3eb-be63-41fc-b042-017c3500aeba (at 192.168.202.14@tcp) reconnecting [ 3472.622937] Lustre: *** cfs_fail_loc=159, val=0*** [ 3472.624866] Lustre: Skipped 4 previous similar messages [ 3479.929688] Lustre: lustre-MDT0000: Client 0d88b89c-dce1-45de-abe6-b9f6dad6ced3 (at 192.168.202.14@tcp) reconnecting [ 3480.833512] Lustre: *** cfs_fail_loc=159, val=0*** [ 3480.835193] Lustre: Skipped 13 previous similar messages [ 3488.122056] Lustre: lustre-MDT0000: Client 0d88b89c-dce1-45de-abe6-b9f6dad6ced3 (at 192.168.202.14@tcp) reconnecting [ 3492.833439] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3492.841063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.854578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3494.487126] Lustre: server umount lustre-OST0000 complete [ 3501.028196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.029242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.042527] Lustre: Skipped 1 previous similar message [ 3502.048283] Lustre: 135028:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317570/real 1761317570] req@00000000dd9ecce8 x1846875218008384/t0(0) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761317577 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3502.052992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.081720] Lustre: 135028:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3502.101021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.111785] Lustre: Skipped 2 previous similar messages [ 3502.944171] Lustre: 135027:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317571/real 1761317571] req@0000000042297a99 x1846875218008768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317578 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3502.958619] Lustre: 135027:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 3507.169050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.174615] Lustre: Skipped 1 previous similar message [ 3508.194772] Lustre: 135026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317576/real 1761317576] req@00000000ad44d96c x1846875218009664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317583 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3508.223791] Lustre: 135026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3512.289163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3512.388963] Lustre: server umount lustre-MDT0000 complete [ 3513.319466] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.343497] LustreError: Skipped 1 previous similar message [ 3515.499954] LustreError: 135694:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317590 with bad export cookie 14325368940310960979 [ 3515.506386] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3515.778946] Lustre: server umount lustre-MDT0001 complete [ 3526.757914] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3529.493842] Key type lgssc unregistered [ 3529.909736] LNet: 140103:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3529.919430] LNet: Removed LNI 192.168.202.114@tcp [ 3530.735495] Key type .llcrypt unregistered [ 3530.738299] Key type ._llcrypt unregistered [ 3555.046255] alg: No test for adler32 (adler32-zlib) [ 3555.801402] Key type ._llcrypt registered [ 3555.802804] Key type .llcrypt registered [ 3555.895348] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3557.159437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3557.558355] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 3557.736594] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3557.739584] LNet: Accept secure, port 988 [ 3559.352159] Key type lgssc registered [ 3560.708790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3568.159088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.526485] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.576981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.873815] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3574.758745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.875775] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.684266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.993596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3587.393856] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3595.026993] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.296293] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3606.148082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.439527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.030780] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3613.615953] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3617.152891] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3619.784782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3630.001365] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3630.782570] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 3633.441783] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3639.269358] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.282670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.298595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.289241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.291420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.292243] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3644.192733] Lustre: server umount lustre-OST0000 complete [ 3649.506813] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3649.509431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.551355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.555684] Lustre: Skipped 1 previous similar message [ 3655.653711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.655915] Lustre: Skipped 2 previous similar messages [ 3660.772530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.780191] Lustre: Skipped 2 previous similar messages [ 3662.304124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3662.399812] Lustre: server umount lustre-MDT0000 complete [ 3665.565953] LustreError: 141571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317740 with bad export cookie 4706638271926324576 [ 3665.576522] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.848691] Lustre: server umount lustre-MDT0001 complete [ 3675.738373] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3678.408845] Key type lgssc unregistered [ 3678.703984] LNet: 145642:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3678.715046] LNet: Removed LNI 192.168.202.114@tcp [ 3679.408858] Key type .llcrypt unregistered [ 3679.411402] Key type ._llcrypt unregistered [ 3698.813833] alg: No test for adler32 (adler32-zlib) [ 3699.571953] Key type ._llcrypt registered [ 3699.575194] Key type .llcrypt registered [ 3699.680141] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3700.207604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3700.697723] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 3700.842954] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3700.846667] LNet: Accept secure, port 988 [ 3702.472561] Key type lgssc registered [ 3703.340158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.942247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.311267] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.374247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.199237] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3716.579823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.708187] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.040664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.632945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.219500] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3736.821581] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3741.910253] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3747.214751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.453486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.603785] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3756.659330] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3758.448753] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3760.751134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3767.850850] Lustre: *** cfs_fail_loc=159, val=0*** [ 3775.341167] Lustre: lustre-MDT0000: Client 74f2a7dd-5653-4afd-85e5-42215b62ec03 (at 192.168.202.14@tcp) reconnecting [ 3781.096187] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3781.099666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.113951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.608531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.622142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3784.571864] Lustre: server umount lustre-OST0000 complete [ 3791.338089] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3791.340650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.371422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.819607] Lustre: server umount lustre-MDT0000 complete [ 3796.781729] LustreError: 147088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317872 with bad export cookie 2385273790761897133 [ 3796.789413] LustreError: 147088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3796.789678] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3797.023395] Lustre: server umount lustre-MDT0001 complete [ 3805.992638] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3808.340920] Key type lgssc unregistered [ 3808.558599] LNet: 150872:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3808.572026] LNet: Removed LNI 192.168.202.114@tcp [ 3809.248752] Key type .llcrypt unregistered [ 3809.251152] Key type ._llcrypt unregistered [ 3827.603128] alg: No test for adler32 (adler32-zlib) [ 3828.355454] Key type ._llcrypt registered [ 3828.357327] Key type .llcrypt registered [ 3828.488871] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 3829.248845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3829.780663] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 3829.973715] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 3829.979565] LNet: Accept secure, port 988 [ 3831.648264] Key type lgssc registered [ 3832.662675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3838.443949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.725099] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3839.766995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.995429] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3845.093695] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.766452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.996096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.713972] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3856.003143] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3859.230313] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3863.156549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.357595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.360073] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 3865.284728] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 3869.962570] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3870.523662] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 3890.445737] Lustre: lustre-OST0000: evicting 22f24616-f04c-422d-b910-527d4a42b571 (at 192.168.202.14@tcp) by administrative request [ 3890.450658] Lustre: Skipped 1 previous similar message [ 3916.772661] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3916.775847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.784067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3916.788026] Lustre: Skipped 1 previous similar message [ 3921.900634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3921.906548] Lustre: Skipped 1 previous similar message [ 3927.011594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.016938] Lustre: Skipped 1 previous similar message [ 3931.104523] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3931.297779] Lustre: server umount lustre-OST0000 complete [ 3936.224988] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3936.232085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.246798] Lustre: Skipped 1 previous similar message [ 3936.250404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.251766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.136701] Lustre: server umount lustre-MDT0000 complete [ 3942.377584] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.383468] LustreError: Skipped 1 previous similar message [ 3942.689742] LustreError: 152314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318018 with bad export cookie 1786106072221061723 [ 3942.693061] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.703197] LustreError: 152314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3942.900969] Lustre: server umount lustre-MDT0001 complete [ 3951.230204] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 3953.040442] Key type lgssc unregistered [ 3953.238451] LNet: 156526:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3953.242472] LNet: Removed LNI 192.168.202.114@tcp [ 3953.804912] Key type .llcrypt unregistered [ 3953.807476] Key type ._llcrypt unregistered [ 4008.833417] alg: No test for adler32 (adler32-zlib) [ 4009.596449] Key type ._llcrypt registered [ 4009.598029] Key type .llcrypt registered [ 4009.679889] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4010.416075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4010.770838] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 4010.944622] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4010.959727] LNet: Accept secure, port 988 [ 4012.632268] Key type lgssc registered [ 4013.432039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4019.699386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.068243] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.130540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.977865] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4026.337860] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.102525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.357545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4032.711933] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4038.114037] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.702910] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4048.105206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.367859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.457285] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4052.056257] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4058.641279] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4060.654497] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 4070.882204] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4070.883645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.896599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4070.908111] Lustre: Skipped 1 previous similar message [ 4073.660596] Lustre: server umount lustre-OST0000 complete [ 4076.004067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.011649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4076.011758] Lustre: Skipped 1 previous similar message [ 4076.016947] Lustre: Skipped 2 previous similar messages [ 4081.121866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.127846] Lustre: Skipped 1 previous similar message [ 4086.243684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4086.246358] Lustre: Skipped 1 previous similar message [ 4090.336154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4090.418324] Lustre: server umount lustre-MDT0000 complete [ 4092.387114] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.398215] LustreError: Skipped 1 previous similar message [ 4093.015489] LustreError: 159062:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318168 with bad export cookie 14915081422820916582 [ 4093.016476] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.026495] LustreError: 159062:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4093.324738] Lustre: server umount lustre-MDT0001 complete [ 4102.686282] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4104.842280] Key type lgssc unregistered [ 4105.044698] LNet: 162743:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4105.053120] LNet: Removed LNI 192.168.202.114@tcp [ 4105.662395] Key type .llcrypt unregistered [ 4105.663898] Key type ._llcrypt unregistered [ 4121.232508] alg: No test for adler32 (adler32-zlib) [ 4121.983586] Key type ._llcrypt registered [ 4121.985846] Key type .llcrypt registered [ 4122.058164] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4122.665719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4123.050854] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 4123.231908] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4123.236340] LNet: Accept secure, port 988 [ 4124.873262] Key type lgssc registered [ 4125.613784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4130.955701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4136.805994] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:03:30 (1761318210) [ 4148.210879] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4154.027432] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4161.071189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.778924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.939798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4178.123804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.715806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.775665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.956331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4184.977955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4185.027949] Lustre: lustre-MDT0000: new disk, initializing [ 4185.110805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.122649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4187.887356] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4195.099532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4195.158609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.225408] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4195.260494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4195.264794] Lustre: Skipped 1 previous similar message [ 4195.323551] Lustre: lustre-MDT0001: new disk, initializing [ 4195.384959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4195.410310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4195.418064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4198.148249] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4204.673550] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4208.713581] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4213.848231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4213.891098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4213.894758] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4214.058737] Lustre: lustre-OST0000: new disk, initializing [ 4214.065175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4214.131443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.028369] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4224.098696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4224.114591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4224.182802] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4234.722876] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4234.726772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.734025] LustreError: Skipped 1 previous similar message [ 4234.735650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4234.751680] Lustre: Skipped 1 previous similar message [ 4234.759810] Lustre: Skipped 1 previous similar message [ 4236.970773] Lustre: server umount lustre-OST0000 complete [ 4239.842454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.846639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.848324] Lustre: Skipped 1 previous similar message [ 4239.855795] Lustre: Skipped 1 previous similar message [ 4244.961482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4244.965581] Lustre: Skipped 1 previous similar message [ 4245.730610] Lustre: server umount lustre-MDT0000 complete [ 4248.079179] LustreError: 167141:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318323 with bad export cookie 5998843730922832397 [ 4248.082374] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.093456] LustreError: 167141:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.289362] Lustre: server umount lustre-MDT0001 complete [ 4255.819764] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4257.361627] Key type lgssc unregistered [ 4257.543916] LNet: 171143:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4257.550070] LNet: Removed LNI 192.168.202.114@tcp [ 4258.044761] Key type .llcrypt unregistered [ 4258.046980] Key type ._llcrypt unregistered [ 4272.344298] alg: No test for adler32 (adler32-zlib) [ 4273.099832] Key type ._llcrypt registered [ 4273.101857] Key type .llcrypt registered [ 4273.190621] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4273.705849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4273.989791] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 4274.105254] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4274.108320] LNet: Accept secure, port 988 [ 4275.728463] Key type lgssc registered [ 4276.386595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4282.237260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.636970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.686466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.232706] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4288.993286] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.091309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.435020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4294.698453] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4300.346977] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4305.033402] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4309.531549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.729708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.817624] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4313.857705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4318.149604] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4325.576112] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4325.592924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.601988] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4325.605700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4325.612399] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 4325.617804] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4326.287761] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4326.298466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.305280] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4326.310208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4326.330588] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 4326.333297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4327.180245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.195228] Lustre: lustre-OST0000: deleting orphan objects from 0x0:121 to 0x0:161 [ 4327.863117] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4327.866182] Lustre: Skipped 1 previous similar message [ 4327.877328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.885814] Lustre: Skipped 1 previous similar message [ 4327.891804] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4327.899425] Lustre: Skipped 1 previous similar message [ 4327.902898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.913470] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 4327.913747] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:225 [ 4327.917525] Lustre: Skipped 1 previous similar message [ 4328.551320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4328.567269] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:225 [ 4329.143762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.149184] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:257 [ 4329.725510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.733995] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:289 [ 4330.288135] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4330.290696] Lustre: Skipped 3 previous similar messages [ 4330.304918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.310749] Lustre: Skipped 3 previous similar messages [ 4330.316247] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4330.319482] Lustre: Skipped 3 previous similar messages [ 4330.322386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.328795] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 4330.332082] Lustre: Skipped 3 previous similar messages [ 4330.332388] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:321 [ 4330.960215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.975868] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:353 [ 4331.520642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.530936] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:385 [ 4339.179077] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4339.180288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.197666] LustreError: Skipped 1 previous similar message [ 4339.232911] Lustre: Skipped 2 previous similar messages [ 4339.243234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4341.729411] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4341.733940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4341.752023] LustreError: Skipped 2 previous similar messages [ 4344.265030] Lustre: server umount lustre-OST0000 complete [ 4346.851309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.858162] Lustre: Skipped 2 previous similar messages [ 4351.972196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4352.935026] Lustre: server umount lustre-MDT0000 complete [ 4356.061734] LustreError: 172591:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318431 with bad export cookie 12939301055770403713 [ 4356.064036] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.095470] LustreError: 172591:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4356.332728] Lustre: server umount lustre-MDT0001 complete [ 4364.365501] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4365.991673] Key type lgssc unregistered [ 4366.214764] LNet: 177069:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4366.222491] LNet: Removed LNI 192.168.202.114@tcp [ 4366.710982] Key type .llcrypt unregistered [ 4366.714494] Key type ._llcrypt unregistered [ 4383.145533] alg: No test for adler32 (adler32-zlib) [ 4383.897574] Key type ._llcrypt registered [ 4383.899269] Key type .llcrypt registered [ 4383.971715] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4384.694060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4384.967870] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 4385.141532] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4385.143961] LNet: Accept secure, port 988 [ 4386.776164] Key type lgssc registered [ 4387.460449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4392.118541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.398803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.452598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.414070] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4398.567938] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.925818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.159758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.912980] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4410.721723] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4414.162721] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4418.367379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.535685] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.613142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.513557] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4422.709284] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:417 [ 4426.768809] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.447234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4431.582412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.590071] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4431.600169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.605740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4431.607149] LustreError: 178554:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4432.867419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.868318] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.872336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.882033] Lustre: Skipped 1 previous similar message [ 4437.991760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4437.995656] Lustre: Skipped 3 previous similar messages [ 4443.114917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4443.117701] Lustre: Skipped 1 previous similar message [ 4447.200157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4447.322723] Lustre: server umount lustre-OST0000 complete [ 4451.608141] LustreError: 178554:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 4451.620581] LustreError: 178554:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4478.834169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.066722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.043134] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4488.378285] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4494.438102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.446917] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.450037] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.114@tcp (at 0@lo) [ 4494.456938] Lustre: lustre-OST0000: deleting orphan objects from 0x0:204 to 0x0:449 [ 4494.470561] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 4494.474325] Lustre: Skipped 1 previous similar message [ 4495.870980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 4500.590939] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f4847e8d000.ost_server_uuid 40 [ 4501.673830] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f4847e8d000.ost_server_uuid in FULL state after 0 sec [ 4505.057636] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4505.066868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.075159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4505.077921] Lustre: Skipped 2 previous similar messages [ 4510.127870] Lustre: server umount lustre-OST0000 complete [ 4515.298650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.301249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.304502] Lustre: Skipped 2 previous similar messages [ 4518.522422] Lustre: server umount lustre-MDT0000 complete [ 4520.425552] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.445584] LustreError: Skipped 1 previous similar message [ 4521.173989] LustreError: 178516:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318596 with bad export cookie 17422461364518368060 [ 4521.178282] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.188806] LustreError: 178516:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4521.474843] Lustre: server umount lustre-MDT0001 complete [ 4530.036933] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 4531.928195] Key type lgssc unregistered [ 4532.089230] LNet: 183827:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4532.098027] LNet: Removed LNI 192.168.202.114@tcp [ 4532.586246] Key type .llcrypt unregistered [ 4532.588270] Key type ._llcrypt unregistered [ 4552.275989] alg: No test for adler32 (adler32-zlib) [ 4553.030691] Key type ._llcrypt registered [ 4553.032399] Key type .llcrypt registered [ 4553.105320] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4553.777726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4554.070600] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 4554.223773] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 4554.227837] LNet: Accept secure, port 988 [ 4555.888117] Key type lgssc registered [ 4556.801794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4563.939787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.308080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.375258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.573280] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4570.595746] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.388725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.695391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4578.980292] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4584.179437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.444749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.003862] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4592.131258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4592.169961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4592.174309] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4592.252485] Lustre: lustre-OST0001: new disk, initializing [ 4592.256434] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4592.301743] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4595.367227] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4595.376456] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 4597.820562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4597.825817] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4603.886058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4608.108206] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4621.797522] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4621.801392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.814613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.844665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.847496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.851453] Lustre: Skipped 1 previous similar message [ 4623.857951] Lustre: Skipped 3 previous similar messages [ 4627.636179] Lustre: server umount lustre-MDT0000 complete [ 4628.962327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.973744] LustreError: Skipped 3 previous similar messages [ 4630.289836] LustreError: 188642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318705 with bad export cookie 7597916535592604331 [ 4630.291655] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.299812] LustreError: 188642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4630.494299] Lustre: server umount lustre-MDT0001 complete [ 4639.200802] Lustre: 189384:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318708/real 1761318708] req@0000000020e1bc83 x1846876442767616/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318714 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4639.234128] Lustre: server umount lustre-OST0000 complete [ 4647.392169] Lustre: 189585:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318716/real 1761318716] req@00000000452543d7 x1846876442768192/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318722 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4647.532142] Lustre: server umount lustre-OST0001 complete [ 4649.221864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4649.292355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4649.869612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4649.900849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.439963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4650.485737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.976952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4651.029212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.631328] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4664.621223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.879103] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.925904] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.271735] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4669.922907] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.452260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.428441] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4679.439524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4679.605186] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4679.608925] Lustre: Skipped 1 previous similar message [ 4680.612374] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.622043] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:484 to 0x0:513 [ 4681.552251] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4685.868807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.728719] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4693.703235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4695.824022] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4708.323375] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4708.336414] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.363120] Lustre: Skipped 1 previous similar message [ 4708.372902] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.598491] Lustre: server umount mylustre-MDT0000 complete [ 4712.252294] LustreError: 194238:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318787 with bad export cookie 7597916535592607040 [ 4712.257179] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.258441] LustreError: 194238:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4721.120135] Lustre: 194746:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318790/real 1761318790] req@0000000061455c06 x1846876442792896/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318796 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4721.149098] Lustre: server umount mylustre-OST0000 complete [ 4721.150929] Lustre: Skipped 1 previous similar message [ 4721.569371] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.575677] Lustre: Skipped 3 previous similar messages [ 4724.772146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4724.822727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.411198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4725.470877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4726.086727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4726.143929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4726.772193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4726.839139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4733.533334] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4738.613061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.882875] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.892070] LustreError: Skipped 1 previous similar message [ 4738.927776] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.931087] Lustre: Skipped 1 previous similar message [ 4741.173688] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4745.416941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.357485] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4750.855416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4752.998554] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4754.086944] Lustre: tfs-OST0000: deleting orphan objects from 0x0:484 to 0x0:545 [ 4757.044081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.656203] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 4759.079445] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4767.181132] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4769.211287] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4781.541117] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.541233] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4781.543981] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.543989] Lustre: Skipped 4 previous similar messages [ 4781.546900] Lustre: Skipped 2 previous similar messages [ 4783.864412] Lustre: server umount tfs-MDT0000 complete [ 4783.865702] Lustre: Skipped 1 previous similar message [ 4785.815480] LustreError: 196387:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318861 with bad export cookie 7597916535592608650 [ 4785.816831] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.820526] LustreError: 196387:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4793.760387] Lustre: 184868:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318862/real 1761318862] req@00000000fb8d7851 x1846876442813440/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318869 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4793.777725] Lustre: 184868:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4793.783331] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.789304] Lustre: Skipped 1 previous similar message [ 4799.285721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4799.329929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4799.988472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4800.039800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.693461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4800.742654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4801.352660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4801.395950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4808.579851] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4814.236558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.524020] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.532206] LustreError: Skipped 2 previous similar messages [ 4814.574510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.578545] Lustre: Skipped 3 previous similar messages [ 4817.025971] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4822.399566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.987550] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4829.777632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.019167] Lustre: lustre-OST0000: deleting orphan objects from 0x0:484 to 0x0:577 [ 4832.559640] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4838.009672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.154810] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 4840.473285] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4844.970828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4847.030750] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4855.304997] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:15:29 (1761318929) [ 4868.577814] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.581847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.590867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4868.593087] Lustre: Skipped 3 previous similar messages [ 4874.488131] Lustre: server umount lustre-MDT0000 complete [ 4874.489609] Lustre: Skipped 3 previous similar messages [ 4876.826624] LustreError: 201684:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318952 with bad export cookie 7597916535592610281 [ 4876.830716] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.837520] LustreError: 201684:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4880.355984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.358611] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4880.368764] LustreError: Skipped 8 previous similar messages [ 4880.371977] Lustre: Skipped 5 previous similar messages [ 4902.445231] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 4906.464988] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 4911.664478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4915.294741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4918.675588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.829940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4925.396567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4925.426389] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4925.426389] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4925.426389] [ 4925.436645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4925.569424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4925.592790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4925.633978] Lustre: lustre-MDT0000: new disk, initializing [ 4925.704258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.707077] Lustre: Skipped 3 previous similar messages [ 4925.716042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4927.831505] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4934.056607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.090139] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 4934.090139] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4934.090139] [ 4934.102032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4934.155531] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4934.196136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4934.199277] Lustre: Skipped 1 previous similar message [ 4934.250873] Lustre: lustre-MDT0001: new disk, initializing [ 4934.321969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4934.327967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4937.191311] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4944.212417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.263328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4944.269318] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4944.397060] Lustre: lustre-OST0000: new disk, initializing [ 4944.400360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4945.485593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4945.489626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4946.614404] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4952.919964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.962366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4952.968239] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4953.051765] Lustre: lustre-OST0001: new disk, initializing [ 4953.054631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4954.910616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4954.916797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4955.514071] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 4961.647714] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:17:15 (1761319035) [ 4967.933619] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:17:21 (1761319041) [ 4980.710371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.710837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.716441] Lustre: Skipped 6 previous similar messages [ 4980.724781] Lustre: Skipped 3 previous similar messages [ 4986.427035] Lustre: server umount lustre-OST0000 complete [ 4986.431380] Lustre: Skipped 3 previous similar messages [ 4990.947477] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4997.307521] LustreError: 207961:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319072 with bad export cookie 7597916535592611800 [ 4997.310949] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.315899] LustreError: 207961:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5010.753106] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5031.392234] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5034.472962] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5042.188403] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5043.212140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.206979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5050.149170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.264977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5056.929836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.541052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.578441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.697413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5060.713850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5060.761617] Lustre: lustre-MDT0000: new disk, initializing [ 5060.821215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.824344] Lustre: Skipped 3 previous similar messages [ 5060.834195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5063.260430] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5069.436642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5069.467464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.639299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5071.571914] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5077.124505] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5080.023782] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5083.455078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5083.488747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.491966] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5083.608311] Lustre: lustre-OST0000: new disk, initializing [ 5083.609835] Lustre: Skipped 1 previous similar message [ 5083.612468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5083.615480] Lustre: Skipped 2 previous similar messages [ 5085.259141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5085.262896] Lustre: Skipped 1 previous similar message [ 5085.265267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5085.575814] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5090.920090] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5095.905319] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.906651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.908582] LustreError: Skipped 1 previous similar message [ 5095.914073] Lustre: Skipped 4 previous similar messages [ 5095.914949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.920771] Lustre: Skipped 9 previous similar messages [ 5110.095309] LustreError: 215142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319185 with bad export cookie 7597916535592613228 [ 5110.098486] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5110.101647] LustreError: 215142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5116.066342] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5117.343108] Key type lgssc unregistered [ 5117.482151] LNet: 219001:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5117.489126] LNet: Removed LNI 192.168.202.114@tcp [ 5117.867500] Key type .llcrypt unregistered [ 5117.869281] Key type ._llcrypt unregistered [ 5129.878977] alg: No test for adler32 (adler32-zlib) [ 5130.630468] Key type ._llcrypt registered [ 5130.632057] Key type .llcrypt registered [ 5130.679797] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5131.194751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5131.368877] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 5131.480118] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 5131.482632] LNet: Accept secure, port 988 [ 5133.096163] Key type lgssc registered [ 5133.573210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5137.242854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.461606] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.494464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.270910] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5143.521424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.122699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.308621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5146.154573] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5149.434942] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5152.082644] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.954746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5155.114622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.025200] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5160.254635] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5160.707514] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 5164.514430] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5164.514577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.526501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.743415] Lustre: server umount lustre-OST0000 complete [ 5170.145285] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5170.150161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.157717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5173.729588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.729918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5173.737835] Lustre: Skipped 1 previous similar message [ 5175.826683] Lustre: server umount lustre-MDT0000 complete [ 5177.625089] LustreError: 220493:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319253 with bad export cookie 4283037664011441094 [ 5177.627768] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5177.630644] LustreError: 220493:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5177.782358] Lustre: server umount lustre-MDT0001 complete [ 5182.488173] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5183.706791] Key type lgssc unregistered [ 5183.853720] LNet: 224215:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5183.857623] LNet: Removed LNI 192.168.202.114@tcp [ 5184.222992] Key type .llcrypt unregistered [ 5184.224848] Key type ._llcrypt unregistered [ 5191.696170] alg: No test for adler32 (adler32-zlib) [ 5192.449381] Key type ._llcrypt registered [ 5192.451480] Key type .llcrypt registered [ 5192.504750] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5192.886916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5193.157220] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 5193.263393] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 5193.267488] LNet: Accept secure, port 988 [ 5194.864124] Key type lgssc registered [ 5195.303218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5199.456100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.992346] LustreError: 225456:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 1: rc = -5 [ 5247.712389] LustreError: 225456:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.114@tcp: failed processing log, type 4: rc = -110 [ 5278.432551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.452489] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5283.788535] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5284.765911] Lustre: server umount lustre-OST0000 complete [ 5291.549872] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5292.778191] Key type lgssc unregistered [ 5292.920488] LNet: 227097:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5292.923992] LNet: Removed LNI 192.168.202.114@tcp [ 5293.280399] Key type .llcrypt unregistered [ 5293.282769] Key type ._llcrypt unregistered [ 5310.014111] alg: No test for adler32 (adler32-zlib) [ 5310.767379] Key type ._llcrypt registered [ 5310.768563] Key type .llcrypt registered [ 5310.812538] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5311.293027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5311.493747] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 5311.588895] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 5311.593060] LNet: Accept secure, port 988 [ 5313.192131] Key type lgssc registered [ 5313.608146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5318.940109] loop0: detected capacity change from 0 to 204800000 [ 5318.995151] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.001994] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.151875] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.158052] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.186532] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.213771] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.234404] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.268675] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.306156] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5319.342934] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5319.750258] loop0: detected capacity change from 0 to 204800000 [ 5319.816689] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5320.134141] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5320.522572] loop0: detected capacity change from 0 to 204800000 [ 5320.722583] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5321.135039] loop0: detected capacity change from 0 to 204800000 [ 5321.334156] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5321.751591] loop0: detected capacity change from 0 to 204800000 [ 5321.767312] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5323.537367] loop0: detected capacity change from 0 to 204800000 [ 5323.550178] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5325.230154] loop0: detected capacity change from 0 to 204800000 [ 5325.248262] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5327.043740] loop0: detected capacity change from 0 to 204800000 [ 5327.061896] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5328.786538] loop0: detected capacity change from 0 to 204800000 [ 5328.838221] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5328.868248] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.899554] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5328.905102] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5328.911649] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5328.918176] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5328.924706] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 5328.930716] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5328.935955] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 5328.941256] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 5328.947840] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5328.956897] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5328.962628] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5328.968313] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5328.974307] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5328.984377] Lustre: 230552:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5338.080272] Lustre: 230830:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319407/real 1761319407] req@00000000f6c742e0 x1846877236494464/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319413 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5338.099833] Lustre: server umount MGS complete [ 5338.563785] loop0: detected capacity change from 0 to 204800000 [ 5338.605770] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.789737] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.816627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.841376] LustreError: 230884:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 5339.253208] loop1: detected capacity change from 0 to 204800000 [ 5339.298879] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5339.340201] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.382608] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5339.388351] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5339.393240] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5339.398991] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 5339.405385] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5339.411230] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 5339.416178] Lustre: 230999:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5339.424746] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5339.518592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.543043] LustreError: 230999:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 5339.939957] loop2: detected capacity change from 0 to 204800000 [ 5339.981194] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.018095] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.021752] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 5340.044927] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5340.049786] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5340.055445] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5340.060431] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5340.065717] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5340.070673] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5340.076078] Lustre: 231083:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5340.137217] Lustre: Found index 0 for lustre-OST0000, updating log [ 5340.189096] LustreError: 230942:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 5340.195158] LustreError: 231083:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 5340.592321] loop3: detected capacity change from 0 to 204800000 [ 5340.661081] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.689849] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.693815] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 5340.713142] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5340.717506] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5340.722418] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5340.728231] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5340.733933] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5340.739337] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5340.745118] Lustre: 231171:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5340.798888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5340.803037] Lustre: Skipped 1 previous similar message [ 5340.812682] LustreError: 231171:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 5343.845372] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 5343.845477] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 5343.845957] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 5343.852610] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 5346.516108] LustreError: 230906:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 4624us: rc = -2 [ 5359.586260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.588681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5359.598402] Lustre: Skipped 2 previous similar messages [ 5359.603390] Lustre: Skipped 3 previous similar messages [ 5364.706042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5364.708636] Lustre: Skipped 3 previous similar messages [ 5369.825259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5369.827835] Lustre: Skipped 3 previous similar messages [ 5372.384708] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5372.428030] Lustre: server umount lustre-MDT0000 complete [ 5372.918873] LustreError: 230891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319448 with bad export cookie 15139352130177715459 [ 5372.920368] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5372.929313] LustreError: 230891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5374.944880] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.945027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.948374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5374.954686] LustreError: Skipped 1 previous similar message [ 5374.960202] Lustre: Skipped 2 previous similar messages [ 5377.953105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.959971] LustreError: Skipped 1 previous similar message [ 5378.977160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5378.980891] Lustre: Skipped 4 previous similar messages [ 5384.160917] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.168689] LustreError: Skipped 3 previous similar messages [ 5387.232232] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5387.367234] Lustre: server umount lustre-MDT0001 complete [ 5402.080298] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5402.098789] Lustre: server umount lustre-OST0000 complete [ 5408.606486] Lustre: server umount lustre-OST0001 complete [ 5411.887211] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:24:45 (1761319485) [ 5425.372971] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5429.271396] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5433.480590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5436.270347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5438.755870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5441.209862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5444.276086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5444.307773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.416876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5444.434586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5444.477732] Lustre: lustre-MDT0000: new disk, initializing [ 5444.522388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.534250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5446.448349] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5452.006876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.035162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.078774] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5452.104591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5452.107370] Lustre: Skipped 1 previous similar message [ 5452.148241] Lustre: lustre-MDT0001: new disk, initializing [ 5452.195321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5452.220725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5452.230037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5454.258278] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5459.156960] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5462.179247] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5465.429970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5465.463926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.467670] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5465.580565] Lustre: lustre-OST0000: new disk, initializing [ 5465.583266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5465.616697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.752159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5466.756744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5467.555553] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5472.590212] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5475.571581] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5484.443920] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5490.896917] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5533.668726] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5533.672983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.701561] LustreError: Skipped 1 previous similar message [ 5533.702590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5533.702596] Lustre: Skipped 2 previous similar messages [ 5533.769831] Lustre: Skipped 1 previous similar message [ 5538.177429] Lustre: server umount lustre-OST0000 complete [ 5544.426080] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.435430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.450593] Lustre: Skipped 1 previous similar message [ 5550.572395] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.805366] LustreError: 238685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319630 with bad export cookie 15139352130177718126 [ 5554.813415] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5554.830249] LustreError: 238685:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5555.120906] Lustre: server umount lustre-MDT0001 complete [ 5555.123612] Lustre: Skipped 1 previous similar message [ 5563.760153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.001857] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5579.232153] Lustre: 240880:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319648/real 1761319648] req@00000000553847f9 x1846877236558208/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319654 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5588.923524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.390599] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.406239] LustreError: Skipped 1 previous similar message [ 5589.419477] LustreError: 241516:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5589.462936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.087183] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5600.298791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.957592] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5609.493602] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5613.277143] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5617.693793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.746283] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5626.703254] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5644.774075] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5644.776612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.788324] LustreError: Skipped 1 previous similar message [ 5644.789041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.789045] Lustre: Skipped 5 previous similar messages [ 5644.817749] Lustre: Skipped 1 previous similar message [ 5649.233149] Lustre: server umount lustre-OST0000 complete [ 5649.234710] Lustre: Skipped 1 previous similar message [ 5651.937257] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5655.009968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.015486] Lustre: Skipped 1 previous similar message [ 5666.274243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5669.136574] LustreError: 241484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319744 with bad export cookie 15139352130177719953 [ 5669.143355] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.144064] LustreError: 241484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5677.024633] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 5678.907329] Key type lgssc unregistered [ 5679.144846] LNet: 245618:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5679.148607] LNet: Removed LNI 192.168.202.114@tcp [ 5679.648401] Key type .llcrypt unregistered [ 5679.650868] Key type ._llcrypt unregistered [ 5709.664874] alg: No test for adler32 (adler32-zlib) [ 5710.438779] Key type ._llcrypt registered [ 5710.440241] Key type .llcrypt registered [ 5710.576485] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 5722.406539] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 5723.345482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5723.781869] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 5724.044496] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 5724.059134] LNet: Accept secure, port 988 [ 5725.805771] Key type lgssc registered [ 5726.878570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.887978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5741.627977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5746.591786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5751.970646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5757.740873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5757.791573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.018123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5759.051800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5759.141823] Lustre: lustre-MDT0000: new disk, initializing [ 5759.249554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.262978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5763.240409] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5772.455668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5772.512450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.575288] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5772.616166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5772.619501] Lustre: Skipped 1 previous similar message [ 5772.685397] Lustre: lustre-MDT0001: new disk, initializing [ 5772.778363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.809495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5772.813790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5776.925312] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5786.493448] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5792.474036] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.040175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5799.099674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.104381] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5799.336528] Lustre: lustre-OST0000: new disk, initializing [ 5799.345909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5799.446381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.636751] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5806.190472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5806.203835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5813.580586] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5819.313843] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5829.275970] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5836.180339] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5847.211602] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5888.482195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.482672] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5888.486605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5888.495488] Lustre: Skipped 1 previous similar message [ 5888.499750] LustreError: Skipped 1 previous similar message [ 5890.470261] Lustre: server umount lustre-OST0000 complete [ 5895.648976] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5895.659246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.678780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5895.691664] Lustre: Skipped 1 previous similar message [ 5898.723327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.723619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.202465] Lustre: server umount lustre-MDT0000 complete [ 5903.668732] LustreError: 249693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319979 with bad export cookie 78334997407257301 [ 5903.674422] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5903.684343] LustreError: 249693:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5904.018661] Lustre: server umount lustre-MDT0001 complete [ 5912.774027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.472372] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5926.368362] Lustre: 255156:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319995/real 1761319995] req@0000000006e0ccad x1846877669606784/t0(0) o251->MGC192.168.202.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320001 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5926.388790] Lustre: server umount MGS complete [ 5934.374959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.686882] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.724374] LustreError: 255791:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5934.789161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.940610] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5940.193713] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.000802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.234713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5947.883977] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5952.966057] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5957.231300] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5961.339682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5961.574672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.002417] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 5968.275655] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5995.491296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5995.491588] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5995.501874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5995.505015] LustreError: Skipped 1 previous similar message [ 5995.511507] Lustre: Skipped 1 previous similar message [ 5997.769120] Lustre: server umount lustre-OST0000 complete [ 6000.610708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.618807] Lustre: Skipped 1 previous similar message [ 6000.624148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6000.629648] Lustre: Skipped 2 previous similar messages [ 6006.692162] Lustre: server umount lustre-MDT0000 complete [ 6008.907455] LustreError: 256566:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320084 with bad export cookie 78334997407259212 [ 6008.909493] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.916982] LustreError: 256566:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6016.267840] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 6017.966551] Key type lgssc unregistered [ 6018.158414] LNet: 259893:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.162865] LNet: Removed LNI 192.168.202.114@tcp [ 6018.623570] Key type .llcrypt unregistered [ 6018.625407] Key type ._llcrypt unregistered [ 6034.925219] alg: No test for adler32 (adler32-zlib) [ 6035.678436] Key type ._llcrypt registered [ 6035.680593] Key type .llcrypt registered [ 6035.754180] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6036.220619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6036.555382] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 6036.691106] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6036.693767] LNet: Accept secure, port 988 [ 6038.320795] Key type lgssc registered [ 6039.149907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6045.144309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.534126] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.576460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.832188] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6051.812399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.161771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.463375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6059.675931] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6066.045638] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6070.531710] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6075.700587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6075.987210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6079.272985] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6085.197131] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6090.539667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6090.603545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.607686] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6090.692838] Lustre: lustre-OST0001: new disk, initializing [ 6090.698238] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6090.769389] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.049557] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6099.022490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6099.034470] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6099.085978] LustreError: 264895:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6100.133169] LustreError: 264895:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6100.139686] LustreError: 264895:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6101.156783] LustreError: 264891:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6101.173564] LustreError: 264891:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6102.994146] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6103.196393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6103.206895] LustreError: 264895:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6103.219329] LustreError: 264895:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 6107.361573] LustreError: 264891:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6107.370516] LustreError: 264891:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 6107.713329] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f4844509000.ost_server_uuid 40 [ 6109.022553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f4844509000.ost_server_uuid in FULL state after 0 sec [ 6113.057093] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f4844509000.ost_server_uuid 40 [ 6114.095829] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f4844509000.ost_server_uuid in FULL state after 0 sec [ 6130.028677] Lustre: lustre-OST0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 6130.150697] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.160040] Lustre: Skipped 1 previous similar message [ 6133.682548] Lustre: server umount lustre-OST0001 complete [ 6135.127254] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.849484] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6138.849568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6138.854958] LustreError: Skipped 1 previous similar message [ 6138.860218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6138.862832] Lustre: Skipped 1 previous similar message [ 6138.875327] Lustre: Skipped 3 previous similar messages [ 6140.386265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.391884] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.392861] Lustre: Skipped 1 previous similar message [ 6140.399675] LustreError: Skipped 2 previous similar messages [ 6143.367058] Lustre: server umount lustre-OST0000 complete [ 6148.577353] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6148.598854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.610240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6150.628369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.343222] Lustre: server umount lustre-MDT0000 complete [ 6155.235232] LustreError: 261437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320230 with bad export cookie 13227626950939435773 [ 6155.236768] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.263792] LustreError: 261437:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6155.518506] Lustre: server umount lustre-MDT0001 complete [ 6163.532722] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 6165.502384] Key type lgssc unregistered [ 6165.732810] LNet: 267113:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6165.744490] LNet: Removed LNI 192.168.202.114@tcp [ 6166.285167] Key type .llcrypt unregistered [ 6166.287170] Key type ._llcrypt unregistered [ 6191.213817] alg: No test for adler32 (adler32-zlib) [ 6191.969430] Key type ._llcrypt registered [ 6191.972664] Key type .llcrypt registered [ 6192.063053] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6192.809288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.002811] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 6193.145584] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6193.149223] LNet: Accept secure, port 988 [ 6194.754790] Key type lgssc registered [ 6195.274064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.301602] loop0: detected capacity change from 0 to 1099511627776 [ 6199.583485] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6199.588363] *** VALIDATE xfs *** [ 6199.596574] XFS (loop0): Mounting V5 Filesystem [ 6199.615336] XFS (loop0): Ending clean mount [ 6200.273376] loop1: detected capacity change from 0 to 18691697672192 [ 6230.548791] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.349019] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 6243.377532] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6248.954489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6252.402352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.801571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.321315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6263.089447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6263.132881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.270459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6264.299590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6264.343681] Lustre: lustre-MDT0000: new disk, initializing [ 6264.428282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.440844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6267.306484] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6273.901759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6273.958839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.025202] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6274.049186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6274.052343] Lustre: Skipped 1 previous similar message [ 6274.094894] Lustre: lustre-MDT0001: new disk, initializing [ 6274.153952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.180025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6274.186048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6276.521436] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6282.326259] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6286.292070] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6290.708338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6290.747340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6290.751693] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6290.908914] Lustre: lustre-OST0000: new disk, initializing [ 6290.911637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6290.966657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.528495] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6298.652746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6298.658281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6299.448988] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6301.859337] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6301.981529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6304.428293] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6304.529571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6308.833292] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6308.837659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.844827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.847096] Lustre: Skipped 1 previous similar message [ 6311.806570] Lustre: server umount lustre-OST0000 complete [ 6313.953618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.954173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6313.967475] Lustre: Skipped 2 previous similar messages [ 6319.074372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.081133] Lustre: Skipped 2 previous similar messages [ 6324.197959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.200434] Lustre: Skipped 1 previous similar message [ 6328.288442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6328.358487] Lustre: server umount lustre-MDT0000 complete [ 6330.341098] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.348425] LustreError: Skipped 1 previous similar message [ 6330.791494] LustreError: 272047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320406 with bad export cookie 2174719009702411246 [ 6330.794690] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.801634] LustreError: 272047:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6330.966819] Lustre: server umount lustre-MDT0001 complete [ 6335.428758] XFS (loop0): Unmounting Filesystem [ 6337.793664] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:40:11 (1761320411) [ 6341.563660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.804627] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.850916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.161878] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6347.234691] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.047429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.267823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6351.495814] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6355.833130] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6358.918444] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6362.782524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.956135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.749957] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6370.974516] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6382.560960] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6382.562693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.568485] LustreError: Skipped 1 previous similar message [ 6382.586886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.591921] Lustre: Skipped 3 previous similar messages [ 6388.273487] Lustre: server umount lustre-OST0000 complete [ 6392.802684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.808879] Lustre: Skipped 1 previous similar message [ 6392.812618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6392.815491] Lustre: Skipped 3 previous similar messages [ 6396.648971] Lustre: server umount lustre-MDT0000 complete [ 6397.926177] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6397.936282] LustreError: Skipped 1 previous similar message [ 6399.020195] LustreError: 276731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320474 with bad export cookie 2174719009702412100 [ 6399.022857] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.025599] LustreError: 276731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6406.881895] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 6408.872540] Key type lgssc unregistered [ 6409.075536] LNet: 280498:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6409.082907] LNet: Removed LNI 192.168.202.114@tcp [ 6409.685778] Key type .llcrypt unregistered [ 6409.688638] Key type ._llcrypt unregistered [ 6427.352940] alg: No test for adler32 (adler32-zlib) [ 6428.110359] Key type ._llcrypt registered [ 6428.112266] Key type .llcrypt registered [ 6428.189944] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 6428.925334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6429.262662] Lustre: Lustre: Build Version: 2.15.7_9_g4830cc0 [ 6429.396398] LNet: Added LNI 192.168.202.114@tcp [8/256/0/180] [ 6429.402381] LNet: Accept secure, port 988 [ 6431.032383] Key type lgssc registered [ 6431.642662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6436.707650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.995505] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.033058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.190204] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6443.492429] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.947035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.191471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6449.902794] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6455.098163] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6458.692846] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6462.444747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.624748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.849350] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6468.703478] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.186936] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.188141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6473.192693] Lustre: Skipped 1 previous similar message [ 6473.196885] Lustre: Skipped 1 previous similar message [ 6475.093541] Lustre: server umount lustre-MDT0001 complete [ 6475.103789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.605848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6478.308275] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.321760] LustreError: Skipped 1 previous similar message [ 6481.041840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.098046] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6481.100683] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6481.122859] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6481.293338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6484.128203] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6486.505068] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6486.517159] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6486.521893] Lustre: Skipped 1 previous similar message [ 6489.069833] LustreError: 285904:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 6489.074363] LustreError: 285904:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6489.079173] LustreError: 285904:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6489.084392] LustreError: 282106:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6489.114458] LustreError: 285907:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -17 [ 6489.118525] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6489.118525] [ 6787.436368] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 6788.578660] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.578854] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6788.588630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6788.597142] Lustre: Skipped 1 previous similar message [ 6792.529882] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 6792.848866] Lustre: server umount lustre-MDT0001 complete [ 6793.698264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.704060] LustreError: Skipped 2 previous similar messages [ 6794.126177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6797.567044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.625947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6797.628854] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6797.653838] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6797.804119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6799.871133] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 6802.915521] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6802.925636] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6802.928872] Lustre: Skipped 1 previous similar message [ 6806.006117] LustreError: 297266:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 6806.012088] LustreError: 297266:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 6806.019732] LustreError: 297266:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6806.029436] LustreError: 297266:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6806.035615] LustreError: 282106:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6806.040925] LustreError: 282106:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 6806.077149] LustreError: 297283:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 6806.080402] LustreError: 297283:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -17 [ 6806.084602] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.114@tcp [ 6806.084602] [ 7104.859855] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.14@tcp (stopping) [ 7104.994510] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.995193] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7104.999777] Lustre: Skipped 1 previous similar message [ 7109.744571] Lustre: server umount lustre-MDT0001 complete [ 7109.971615] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.977922] LustreError: Skipped 1 previous similar message [ 7111.109928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7114.869324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7114.922403] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7114.924923] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7114.945186] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7115.082260] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7116.986151] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 7120.358366] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7120.366083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7120.369982] Lustre: Skipped 1 previous similar message [ 7122.923449] LustreError: 309024:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7122.929443] LustreError: 309024:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7122.934570] LustreError: 309024:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7122.940916] LustreError: 282106:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7122.944135] LustreError: 282106:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7122.966364] LustreError: 309030:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7122.973560] LustreError: 309030:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.114@tcp: cfg command failed: rc = -114 [ 7122.978198] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7122.978198] [ 7432.673591] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7432.676957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.681119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.681792] Lustre: Skipped 3 previous similar messages [ 7432.684989] Lustre: Skipped 3 previous similar messages [ 7437.917577] Lustre: server umount lustre-MDT0000 complete [ 7439.437096] LustreError: 308394:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321514 with bad export cookie 10319415388647061945 [ 7439.440310] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.442281] LustreError: 308394:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7442.914625] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.914768] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.920326] LustreError: Skipped 2 previous similar messages [ 7445.637505] Lustre: server umount lustre-MDT0001 complete [ 7451.124666] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7454.763674] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7459.074759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7462.022334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7464.867253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7467.713183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7470.493309] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:59:04 (1761321544) [ 7473.335321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7473.365786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.465883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7473.483610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7473.527303] Lustre: lustre-MDT0000: new disk, initializing [ 7473.567754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.576866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7475.429791] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 7480.334440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7480.364160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.408972] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7480.430086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7480.432471] Lustre: Skipped 1 previous similar message [ 7480.470299] Lustre: lustre-MDT0001: new disk, initializing [ 7480.511263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7480.525899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7480.530034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7482.308990] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 7486.649845] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7488.915288] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7491.554672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7491.583153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.587514] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7491.690123] Lustre: lustre-OST0000: new disk, initializing [ 7491.692521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7491.721792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.591451] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_default_debug -1 all 8 [ 7495.169439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7495.174356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7498.243582] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7505.378341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.379135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7505.383839] Lustre: Skipped 2 previous similar messages [ 7505.386683] Lustre: Skipped 7 previous similar messages [ 7505.958408] Lustre: server umount lustre-MDT0000 complete [ 7505.960033] Lustre: Skipped 1 previous similar message [ 7507.271292] LustreError: 323076:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321582 with bad export cookie 10319415388647063856 [ 7507.274172] LustreError: 323945:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 7507.274826] LustreError: 166-1: MGC192.168.202.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7507.275588] LustreError: 323076:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7507.280285] LustreError: 323945:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 7507.290169] LustreError: 323945:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 7515.104180] Lustre: 326333:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321584/real 1761321584] req@000000003b68d239 x1846878408979136/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761321590 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7526.861686] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing set_hostid [ 7530.341202] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing load_modules_local [ 7535.086600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7538.457418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7541.201866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7543.924067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.634846] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7281 sec ========== 12:00:18 (1761321618) [ 7563.067318] Lustre: DEBUG MARKER: oleg214-server.virtnet: executing unload_modules_local [ 7564.276048] Key type lgssc unregistered [ 7564.429333] LNet: 331432:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.433935] LNet: Removed LNI 192.168.202.114@tcp [ 7564.800431] Key type .llcrypt unregistered [ 7564.802098] Key type ._llcrypt unregistered