[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477735316 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010056] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021772] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026308] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.031511] *** VALIDATE proc *** [ 0.032219] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034251] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037034] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.041811] debug: unmapping init [mem 0xffffffff89859000-0xffffffff89860fff] [ 0.043843] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044644] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.046958] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.047955] ... max period: 00007fffffffffff [ 0.048019] ... fixed-purpose events: 3 [ 0.048912] ... event mask: 000000070000000f [ 0.049282] rcu: Hierarchical SRCU implementation. [ 0.051143] smp: Bringing up secondary CPUs ... [ 0.052406] x86: Booting SMP configuration: [ 0.053030] .... node #0, CPUs: #1 #2 #3 [ 0.056678] smp: Brought up 1 node, 4 CPUs [ 0.058017] smpboot: Max logical packages: 1 [ 0.059017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161030] node 0 deferred pages initialised in 101ms [ 0.163167] devtmpfs: initialized [ 0.164449] x86/mm: Memory block size: 128MB [ 0.168222] gcov: version magic: 0x41383552 [ 0.172385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.173111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.176515] pinctrl core: initialized pinctrl subsystem [ 0.178522] [ 0.179018] ************************************************************* [ 0.182022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185025] ** ** [ 0.187019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.190024] ** ** [ 0.193022] ** This means that this kernel is built to expose internal ** [ 0.195082] ** IOMMU data structures, which may compromise security on ** [ 0.197023] ** your system. ** [ 0.200021] ** ** [ 0.202018] ** If you see this message and you are not debugging the ** [ 0.205041] ** kernel, report this immediately to your vendor! ** [ 0.207020] ** ** [ 0.210021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212026] ************************************************************* [ 0.215696] NET: Registered protocol family 16 [ 0.218461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.224072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.227044] cpuidle: using governor menu [ 0.228788] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.231707] PCI: Using configuration type 1 for base access [ 0.234158] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.244125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.247038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.250195] cryptd: max_cpu_qlen set to 1000 [ 0.253359] ACPI: Added _OSI(Module Device) [ 0.255015] ACPI: Added _OSI(Processor Device) [ 0.257012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.258014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.263141] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.269669] ACPI: Interpreter enabled [ 0.271089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273021] ACPI: Using IOAPIC for interrupt routing [ 0.274155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.278459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.288589] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.302995] acpiphp: Slot [2] registered [ 0.305254] acpiphp: Slot [5] registered [ 0.306192] acpiphp: Slot [6] registered [ 0.308193] acpiphp: Slot [7] registered [ 0.309205] acpiphp: Slot [8] registered [ 0.311143] acpiphp: Slot [9] registered [ 0.312201] acpiphp: Slot [10] registered [ 0.314239] acpiphp: Slot [3] registered [ 0.316169] acpiphp: Slot [4] registered [ 0.317156] acpiphp: Slot [11] registered [ 0.319156] acpiphp: Slot [12] registered [ 0.320152] acpiphp: Slot [13] registered [ 0.322152] acpiphp: Slot [14] registered [ 0.323195] acpiphp: Slot [15] registered [ 0.325189] acpiphp: Slot [16] registered [ 0.327146] acpiphp: Slot [17] registered [ 0.328171] acpiphp: Slot [18] registered [ 0.330191] acpiphp: Slot [19] registered [ 0.331143] acpiphp: Slot [20] registered [ 0.333176] acpiphp: Slot [21] registered [ 0.335200] acpiphp: Slot [22] registered [ 0.337137] acpiphp: Slot [23] registered [ 0.338229] acpiphp: Slot [24] registered [ 0.340135] acpiphp: Slot [25] registered [ 0.342128] acpiphp: Slot [26] registered [ 0.343131] acpiphp: Slot [27] registered [ 0.344098] acpiphp: Slot [28] registered [ 0.346178] acpiphp: Slot [29] registered [ 0.347118] acpiphp: Slot [30] registered [ 0.349133] acpiphp: Slot [31] registered [ 0.351081] PCI host bridge to bus 0000:00 [ 0.352019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.357021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.358018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.360020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363098] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.366198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.370087] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372675] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.383877] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.390063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.392020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.396017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.399628] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.402475] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.405048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407763] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.413016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.429015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468652] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.521731] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.552763] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.581394] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.587016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.602000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.607918] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.627020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.645017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.656846] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.659319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661310] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.663302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.664195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.669033] iommu: Default domain type: Passthrough [ 0.670344] SCSI subsystem initialized [ 0.671106] ACPI: bus type USB registered [ 0.672091] usbcore: registered new interface driver usbfs [ 0.673088] usbcore: registered new interface driver hub [ 0.674076] usbcore: registered new device driver usb [ 0.675113] pps_core: LinuxPPS API ver. 1 registered [ 0.677009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678052] PTP clock support registered [ 0.680168] EDAC MC: Ver: 3.0.0 [ 0.682115] PCI: Using ACPI for IRQ routing [ 0.684831] NetLabel: Initializing [ 0.686013] NetLabel: domain hash size = 128 [ 0.687011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.689077] NetLabel: unlabeled traffic allowed by default [ 0.691096] vgaarb: loaded [ 0.692294] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.700008] clocksource: Switched to clocksource kvm-clock [ 0.814066] VFS: Disk quotas dquot_6.6.0 [ 0.815909] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.818686] *** VALIDATE ramfs *** [ 0.820055] *** VALIDATE hugetlbfs *** [ 0.821823] pnp: PnP ACPI init [ 0.824474] pnp: PnP ACPI: found 6 devices [ 0.842326] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.845788] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.848184] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850578] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.853031] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.855523] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.858526] NET: Registered protocol family 2 [ 0.860968] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.865822] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.869709] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.875138] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.877899] TCP: Hash tables configured (established 65536 bind 65536) [ 0.880319] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.883609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.886086] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889076] NET: Registered protocol family 1 [ 0.892760] RPC: Registered named UNIX socket transport module. [ 0.895068] RPC: Registered udp transport module. [ 0.896487] RPC: Registered tcp transport module. [ 0.897669] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.898981] NET: Registered protocol family 44 [ 0.900099] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.901341] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.902810] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.904330] PCI: CLS 0 bytes, default 64 [ 0.905392] Unpacking initramfs... [ 2.263566] debug: unmapping init [mem 0xffff96743cc54000-0xffff96743ffbffff] [ 2.267325] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.269427] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.272017] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.756441] Initialise system trusted keyrings [ 2.758025] Key type blacklist registered [ 2.761055] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.769518] zbud: loaded [ 2.772532] *** VALIDATE nfs *** [ 2.773591] *** VALIDATE nfs4 *** [ 2.774871] pstore: using deflate compression [ 2.777638] Platform Keyring initialized [ 2.886102] NET: Registered protocol family 38 [ 2.887952] Key type asymmetric registered [ 2.890636] Asymmetric key parser 'x509' registered [ 2.892697] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.896078] io scheduler mq-deadline registered [ 2.897923] io scheduler kyber registered [ 2.899607] io scheduler bfq registered [ 2.901611] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.904406] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.907374] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.910318] ACPI: Power Button [PWRF] [ 2.915281] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.920847] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.929624] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.935615] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.946156] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.973604] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.998775] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.003389] Non-volatile memory driver v1.3 [ 3.005040] Linux agpgart interface v0.103 [ 3.039213] virtio_blk virtio1: [vda] 145856 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.041848] vda: detected capacity change from 0 to 74678272 [ 3.054790] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.056540] vdb: detected capacity change from 0 to 1073741824 [ 3.069436] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.071833] vdc: detected capacity change from 0 to 2621440000 [ 3.084953] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.088013] vdd: detected capacity change from 0 to 2621440000 [ 3.101474] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.104262] vde: detected capacity change from 0 to 4294967296 [ 3.118316] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.121207] vdf: detected capacity change from 0 to 4294967296 [ 3.128969] libphy: Fixed MDIO Bus: probed [ 3.134593] usbcore: registered new interface driver usbserial_generic [ 3.135993] usbserial: USB Serial support registered for generic [ 3.137226] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.141474] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.143055] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.144763] mousedev: PS/2 mouse device common for all mice [ 3.147469] rtc_cmos 00:05: RTC can wake from S4 [ 3.149117] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.150663] rtc_cmos 00:05: registered as rtc0 [ 3.153604] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.155303] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.156354] intel_pstate: CPU model not supported [ 3.160121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.162923] hid: raw HID events driver (C) Jiri Kosina [ 3.164298] usbcore: registered new interface driver usbhid [ 3.166114] usbhid: USB HID core driver [ 3.167460] drop_monitor: Initializing network drop monitor service [ 3.169447] Initializing XFRM netlink socket [ 3.170971] NET: Registered protocol family 10 [ 3.173816] Segment Routing with IPv6 [ 3.175203] NET: Registered protocol family 17 [ 3.176924] mpls_gso: MPLS GSO support [ 3.182977] RAS: Correctable Errors collector initialized. [ 3.185252] AVX version of gcm_enc/dec engaged. [ 3.186939] AES CTR mode by8 optimization enabled [ 3.263391] sched_clock: Marking stable (3263370977, 0)->(4172803380, -909432403) [ 3.266876] registered taskstats version 1 [ 3.268945] Loading compiled-in X.509 certificates [ 3.270319] zswap: loaded using pool lzo/zbud [ 3.294142] Key type big_key registered [ 3.307792] Key type encrypted registered [ 3.309096] ima: No TPM chip found, activating TPM-bypass! [ 3.311170] ima: Allocated hash algorithm: sha1 [ 3.313067] ima: No architecture policies found [ 3.314970] evm: Initialising EVM extended attributes: [ 3.316872] evm: security.selinux [ 3.318179] evm: security.ima [ 3.319163] evm: security.capability [ 3.320382] evm: HMAC attrs: 0x1 [ 3.322725] rtc_cmos 00:05: setting system clock to 2026-08-15 12:37:10 UTC (1786797430) [ 3.329371] debug: unmapping init [mem 0xffffffff8a803000-0xffffffff8a9fffff] [ 3.332763] debug: unmapping init [mem 0xffffffff89582000-0xffffffff89858fff] [ 3.341072] Write protecting the kernel read-only data: 28672k [ 3.344410] debug: unmapping init [mem 0xffffffff87c03000-0xffffffff87dfffff] [ 3.347217] debug: unmapping init [mem 0xffffffff88514000-0xffffffff885fffff] [ 3.380937] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.389360] systemd[1]: Detected virtualization kvm. [ 3.391233] systemd[1]: Detected architecture x86-64. [ 3.393030] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.416849] systemd[1]: No hostname configured. [ 3.418607] systemd[1]: Set hostname to . [ 3.420837] random: systemd: uninitialized urandom read (16 bytes read) [ 3.423321] systemd[1]: Initializing machine ID from random generator. [ 3.459907] random: ln: uninitialized urandom read (6 bytes read) [ 3.550116] random: systemd: uninitialized urandom read (16 bytes read) [ 3.553076] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.561964] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.569322] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.124141] device-mapper: uevent: version 1.0.3 [ 4.126118] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.774248] virtio_net virtio0 ens2: renamed from eth0 [ 4.792817] random: fast init done [ 4.834055] scsi host0: ata_piix [ 4.837850] scsi host1: ata_piix [ 4.839352] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.841755] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.246727] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.836386] random: crng init done [ 9.837817] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.211505] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.326172] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.571952] SELinux: Disabled at runtime. [ 11.633565] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.642932] systemd[1]: Detected virtualization kvm. [ 11.646108] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.109044] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.111894] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.120158] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.124144] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.127322] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.135589] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.146553] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 12.262551] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.581587] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.845324] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.862469] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.004344] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.027863] EDAC sbridge: Ver: 1.1.2 [ 14.493242] Key type dns_resolver registered [ 14.796201] NFS: Registering the id_resolver key type [ 14.798196] Key type id_resolver registered [ 14.799973] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-server login: [ 40.094695] spl: loading out-of-tree module taints kernel. [ 40.313519] hrtimer: interrupt took 3680348 ns [ 48.037325] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.726786] Key type ._llcrypt registered [ 66.737140] Key type .llcrypt registered [ 66.895164] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_hostid [ 91.133788] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 93.254252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.276438] alg: No test for adler32 (adler32-zlib) [ 95.061747] Lustre: Lustre: Build Version: 2.17.55_1_g696fb08 [ 96.448821] LNet: Added LNI 192.168.202.110@tcp [8/256/0/180] [ 98.319341] Key type lgssc registered [ 101.354428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.621524] vdc: vdc1 vdc9 [ 127.491855] vdd: vdd1 vdd9 [ 127.507295] vdd: vdd1 vdd9 [ 139.627279] vde: vde1 vde9 [ 153.452210] vdf: vdf1 vdf9 [ 176.310839] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_modules_local [ 186.274786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.561788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.841192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.955145] Lustre: lustre-MDT0000: new disk, initializing [ 188.532713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.604331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.367458] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.140037] Lustre: 6398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.245061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.249309] Lustre: Skipped 1 previous similar message [ 204.347847] Lustre: lustre-MDT0001: new disk, initializing [ 204.749397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.849141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.859512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.409949] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.620565] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 221.435927] Lustre: lustre-OST0000: new disk, initializing [ 221.439688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.622642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.367227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.381521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.755822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.757887] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.183897] Lustre: lustre-OST0001: new disk, initializing [ 242.187529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.307878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.307582] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.586315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.607924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.994323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.415065] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.536861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.038829] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.519168] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing yml_node [ 288.732680] Lustre: DEBUG MARKER: Client: 2.17.55.1 [ 290.858693] Lustre: DEBUG MARKER: MDS: 2.17.55.1 [ 293.419886] Lustre: DEBUG MARKER: OSS: 2.17.55.1 [ 295.040562] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Aug 15 08:42:01 EDT 2026 [ 312.511537] Lustre: DEBUG MARKER: excepting tests: 110f 131b 59 36 [ 314.473205] Lustre: DEBUG MARKER: === replay-single: start setup 08:42:21 (1786797741) === [ 320.220585] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing check_config_client /mnt/lustre [ 341.787931] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 346.238288] Lustre: 12686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 350.882500] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 355.217600] Lustre: DEBUG MARKER: === replay-single: finish setup 08:43:01 (1786797781) === [ 358.324311] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 08:43:05 (1786797785) [ 359.868961] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 359.870940] LustreError: 7240:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9674bb317100 x1873592999590784/t4294967300(0) o1000->lustre-MDT0000-mdtlov_UUID@0@lo:452/0 lens 264/4320 e 0 to 0 dl 1786797797 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up1-0.0' uid:0 gid:0 projid:4294967295 [ 361.946334] Lustre: Failing over lustre-MDT0001 [ 362.217168] Lustre: server umount lustre-MDT0001 complete [ 363.492221] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.500685] Lustre: Skipped 1 previous similar message [ 368.609181] LustreError: 7240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.624785] LustreError: 7240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 371.465924] LustreError: 6406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.736131] LustreError: 6411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.755657] LustreError: 6411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 375.271929] Lustre: 7237:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786797786/real 1786797786] req@ffff9674bb316d80 x1873592999590784/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 264/4320 e 0 to 1 dl 1786797802 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up1-0.0' uid:0 gid:0 projid:4294967295 [ 376.591585] LustreError: 6407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.123431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 381.708452] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 385.439379] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 385.536218] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 385.663855] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 395.274447] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 396.939451] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 407.272434] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 08:43:53 (1786797833) [ 408.377610] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 408.388652] LustreError: 6407:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9674afdd2a00 x1873592975411200/t4294967364(0) o36->3b785ab9-7be2-496e-8a36-d29e8b915491@192.168.202.10@tcp:541/0 lens 560/536 e 0 to 0 dl 1786797886 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 410.288158] Lustre: Failing over lustre-MDT0000 [ 410.593084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 410.596155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.614492] Lustre: Skipped 1 previous similar message [ 410.626971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 410.698499] Lustre: server umount lustre-MDT0000 complete [ 411.105156] LustreError: 6407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.128272] LustreError: 6407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 421.346773] LustreError: 6406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.370752] LustreError: 6406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 428.007221] Lustre: 3455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786797838/real 1786797838] req@ffff967489ba0a80 x1873592999633536/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786797854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 428.019346] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.244187] LustreError: 6411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.260182] LustreError: 6411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 438.944524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.042476] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 443.767861] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 443.884451] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 443.897306] Lustre: Skipped 2 previous similar messages [ 443.976882] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 444.020055] Lustre: 9432:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff96738210ca80 x1873592975411200/t4294967364(0) o36->3b785ab9-7be2-496e-8a36-d29e8b915491@192.168.202.10@tcp:577/0 lens 560/2880 e 0 to 0 dl 1786797922 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 444.065146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6 to 0x280000401:33) [ 444.114913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6 to 0x2c0000401:33) [ 453.152692] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.734504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 465.298831] Lustre: DEBUG MARKER: == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 08:44:51 (1786797891) [ 468.765066] LustreError: 15970:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 469.568547] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 471.554240] Lustre: Failing over lustre-MDT0001 [ 472.038369] Lustre: server umount lustre-MDT0001 complete [ 474.609470] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.614505] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 474.625986] LustreError: 13548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.633217] Lustre: Skipped 5 previous similar messages [ 481.743705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.854816] Lustre: lustre-MDT0001: Aborting MDT recovery [ 481.888857] LustreError: 16576:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 1, retries 0, failed: rc = -108 [ 482.941190] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 486.912758] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 486.927457] Lustre: Skipped 3 previous similar messages [ 487.512148] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [ 487.741247] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [ 487.926490] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 487.999069] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:46 to 0x280000400:65) [ 488.006386] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:46 to 0x2c0000400:65) [ 488.308961] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 503.757119] Lustre: Failing over lustre-MDT0001 [ 504.118596] Lustre: server umount lustre-MDT0001 complete [ 505.319060] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.331176] Lustre: Skipped 2 previous similar messages [ 522.630892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 523.014717] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 527.131188] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 527.850807] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 527.863079] Lustre: Skipped 2 previous similar messages [ 527.982450] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 528.042939] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:97) [ 528.043353] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:70 to 0x2c0000400:97) [ 537.218939] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 538.820841] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 547.502505] Lustre: DEBUG MARKER: == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 08:46:14 (1786797974) [ 563.059393] Lustre: Failing over lustre-MDT0000 [ 563.292510] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 563.306621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.322727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.577188] Lustre: server umount lustre-MDT0000 complete [ 563.688863] LustreError: 6405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.723996] LustreError: 6405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 571.191607] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.724765] Lustre: lustre-MDT0000: Aborting client recovery [ 571.729984] LustreError: 19077:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 571.734971] Lustre: 19113:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 571.742324] Lustre: 19113:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3b785ab9-7be2-496e-8a36-d29e8b915491@ [ 571.754094] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 571.811572] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 571.900408] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 572.013427] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 572.037326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 577.003385] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 577.013943] Lustre: Skipped 2 previous similar messages [ 577.021734] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 577.126965] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 597.908581] Lustre: DEBUG MARKER: == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 08:47:04 (1786798024) [ 600.757225] LustreError: 20019:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 601.731357] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 604.326440] LustreError: 20215:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 605.059668] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 607.215650] Lustre: Failing over lustre-MDT0000 [ 607.411664] Lustre: server umount lustre-MDT0000 complete [ 607.716616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.732493] Lustre: Skipped 3 previous similar messages [ 611.127974] LustreError: 6392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786798038 with bad export cookie 14690694427229647480 [ 611.132655] Lustre: Failing over lustre-MDT0001 [ 611.132959] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.144601] LustreError: 6392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 611.461530] Lustre: server umount lustre-MDT0001 complete [ 636.330406] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbdfd4bf36d57836 [ 636.340048] Lustre: MGC192.168.202.110@tcp: Connection restored to 0@lo (at 0@lo) [ 636.347190] Lustre: Skipped 3 previous similar messages [ 636.794753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.801341] Lustre: Skipped 1 previous similar message [ 636.892542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 637.346241] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.348341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 637.365329] Lustre: Skipped 2 previous similar messages [ 637.736174] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 638.817036] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 641.734816] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 642.235899] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 644.230244] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 644.254299] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:70 to 0x2c0000400:129) [ 644.258781] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:70 to 0x280000400:129) [ 650.692983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 650.695321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 655.766819] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.218460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 658.608264] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 667.099648] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 08:48:13 (1786798093) [ 669.152292] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798040/real 1786798040] req@ffff9674bad4a680 x1873592999970432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786798096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 670.553732] LustreError: 22669:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 671.369562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 673.760820] Lustre: 3454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798044/real 1786798044] req@ffff96738c29d180 x1873592999970944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786798100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 673.779849] Lustre: 3454:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 682.463460] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798053/real 1786798053] req@ffff9674bad4b800 x1873592999971712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786798109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 682.493813] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 692.705116] Lustre: 3454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798063/real 1786798063] req@ffff9674bc183480 x1873592999972864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786798119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 692.770018] Lustre: 3454:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 692.787662] Lustre: Failing over lustre-MDT0000 [ 693.149965] Lustre: server umount lustre-MDT0000 complete [ 695.266702] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.269178] LustreError: 22017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.294888] Lustre: Skipped 4 previous similar messages [ 695.333017] LustreError: 22017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 700.717921] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.373463] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 701.376044] Lustre: lustre-MDT0000: Aborting client recovery [ 701.378679] LustreError: 23256:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 701.383177] Lustre: 23290:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 701.385945] Lustre: 23290:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 701.414426] Lustre: 23290:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 701.419512] Lustre: 23290:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 701.423606] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 701.462746] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000013a0:0x1:0x0] [ 701.522332] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240001b71:0x1:0x0] [ 701.585100] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:641) [ 701.594576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:641) [ 705.830224] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 706.537471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 706.549411] Lustre: Skipped 6 previous similar messages [ 706.557520] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 790.210249] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 08:50:16 (1786798216) [ 791.666267] Lustre: *** cfs_fail_loc=159, val=0*** [ 791.673990] Lustre: Skipped 1 previous similar message [ 846.212338] Lustre: lustre-MDT0000: Client 3b785ab9-7be2-496e-8a36-d29e8b915491 (at 192.168.202.10@tcp) reconnecting [ 854.753386] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 08:51:21 (1786798281) [ 856.159331] Lustre: *** cfs_fail_loc=15a, val=0*** [ 911.625488] Lustre: lustre-MDT0001: Client 3b785ab9-7be2-496e-8a36-d29e8b915491 (at 192.168.202.10@tcp) reconnecting [ 911.731203] Lustre: 21243:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff96738224e300 x1873592978384000/t21474836525(0) o36->3b785ab9-7be2-496e-8a36-d29e8b915491@192.168.202.10@tcp:288/0 lens 488/3152 e 0 to 0 dl 1786798388 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:4294967295 [ 911.769432] Lustre: 21243:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 6 previous similar messages [ 919.510853] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 08:52:26 (1786798346) [ 922.668484] LustreError: 24921:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 923.619877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 924.678430] Lustre: *** cfs_fail_loc=15a, val=0*** [ 924.688811] Lustre: Skipped 6 previous similar messages [ 928.811886] Lustre: Failing over lustre-MDT0000 [ 929.090446] Lustre: server umount lustre-MDT0000 complete [ 931.808214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.815361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 931.824982] Lustre: Skipped 1 previous similar message [ 947.168708] Lustre: 3454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798358/real 1786798358] req@ffff9674891ab100 x1873593000569984/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786798374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 947.190244] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.289991] LustreError: 21246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.326085] LustreError: 21246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 957.702072] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (not set up) [ 958.061410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.068625] Lustre: Skipped 2 previous similar messages [ 958.218494] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 958.237116] Lustre: Skipped 2 previous similar messages [ 959.542497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 959.548364] Lustre: Skipped 1 previous similar message [ 963.050476] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 963.064412] Lustre: Skipped 3 previous similar messages [ 963.162361] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 963.172109] Lustre: Skipped 1 previous similar message [ 963.220445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1149 to 0x280000401:1185) [ 963.221063] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1149 to 0x2c0000401:1185) [ 963.978685] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 974.097831] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.657310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 984.738139] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [ 986.645586] Lustre: *** cfs_fail_loc=15a, val=0*** [ 986.657261] Lustre: Skipped 6 previous similar messages [ 990.761531] Lustre: Failing over lustre-MDT0001 [ 990.998904] Lustre: server umount lustre-MDT0001 complete [ 993.782988] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.796030] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1010.965729] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1016.034288] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1016.055383] Lustre: 21246:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9674bc056a00 x1873592978449664/t21474836569(0) o36->3b785ab9-7be2-496e-8a36-d29e8b915491@192.168.202.10@tcp:393/0 lens 488/3152 e 0 to 0 dl 1786798493 ref 1 fl Interpret:/202/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:4294967295 [ 1016.106420] Lustre: 21246:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 1016.110738] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:161) [ 1016.129547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:161) [ 1016.372779] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1028.317520] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1030.296930] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1039.354423] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 08:54:26 (1786798466) [ 1044.080085] Lustre: Failing over lustre-MDT0000 [ 1044.524992] Lustre: server umount lustre-MDT0000 complete [ 1046.500014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1062.871095] Lustre: 3452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798473/real 1786798473] req@ffff9674bc054380 x1873593000644736/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786798489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1062.889536] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1073.121448] LustreError: 3451:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff967388c68e00 x1873593000654208/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1074.201691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1074.603731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1079.292986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1079.310539] Lustre: Skipped 6 previous similar messages [ 1079.513859] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1079.572512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1217) [ 1079.573729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1217) [ 1079.864960] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1090.082477] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1091.651867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1100.368384] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 08:55:27 (1786798527) [ 1103.616345] LustreError: 29439:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1104.586477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1106.810959] Lustre: Failing over lustre-MDT0000 [ 1107.202033] Lustre: server umount lustre-MDT0000 complete [ 1108.547889] LustreError: 6392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786798535 with bad export cookie 14690694427229850914 [ 1108.568825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.592829] Lustre: Skipped 8 previous similar messages [ 1125.337401] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.926554] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1130.309711] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1131.196923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1249) [ 1131.197933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1249) [ 1140.408938] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.074370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1151.143042] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 08:56:17 (1786798577) [ 1154.048427] LustreError: 30975:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1154.863392] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1157.064819] Lustre: Failing over lustre-MDT0000 [ 1157.097307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 1157.316081] Lustre: server umount lustre-MDT0000 complete [ 1174.051368] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.849236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1179.631708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1179.647042] Lustre: Skipped 1 previous similar message [ 1179.723323] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1189.790847] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1192.105259] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:57 [ 1197.319391] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:52 [ 1202.439537] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:47 [ 1207.561271] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:41 [ 1212.708202] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:36 [ 1222.943730] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:26 [ 1222.974033] Lustre: Skipped 1 previous similar message [ 1243.398982] Lustre: lustre-MDT0000: Denying connection for new client 9f7d1751-436f-40b4-b443-6eadd3df8850 (at 192.168.202.10@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:06 [ 1243.429500] Lustre: Skipped 3 previous similar messages [ 1249.500191] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1249.504820] Lustre: 31559:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3b785ab9-7be2-496e-8a36-d29e8b915491@ [ 1249.518629] Lustre: 31559:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1249.535162] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1249.606607] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 1249.612027] Lustre: Skipped 1 previous similar message [ 1249.615573] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1249.618283] Lustre: Skipped 10 previous similar messages [ 1249.672478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1281) [ 1249.677213] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1281) [ 1264.147110] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 08:58:10 (1786798690) [ 1267.157975] LustreError: 32510:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 1267.974144] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1270.344606] Lustre: Failing over lustre-MDT0001 [ 1270.565410] Lustre: server umount lustre-MDT0001 complete [ 1271.775678] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1288.952991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.955526] Lustre: Skipped 4 previous similar messages [ 1289.056121] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1294.032521] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:193) [ 1294.032527] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:193) [ 1294.683983] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1305.480771] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1307.091955] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1316.075037] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 08:59:03 (1786798743) [ 1320.309789] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1322.775530] Lustre: Failing over lustre-MDT0001 [ 1323.323284] Lustre: server umount lustre-MDT0001 complete [ 1348.365292] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1349.088439] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1349.100828] Lustre: Skipped 1 previous similar message [ 1358.247773] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1360.737590] Lustre: lustre-MDT0001: Denying connection for new client 912d68fc-778b-4ad7-a515-4b88ee330731 (at 192.168.202.10@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 1360.754959] Lustre: Skipped 1 previous similar message [ 1419.503900] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1419.509974] Lustre: 34605:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 9f7d1751-436f-40b4-b443-6eadd3df8850@ [ 1419.527966] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1419.636669] Lustre: lustre-MDT0001: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 1419.640535] Lustre: Skipped 1 previous similar message [ 1419.663941] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:225) [ 1419.664144] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:225) [ 1430.360688] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 09:00:57 (1786798857) [ 1433.283878] LustreError: 35540:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 1433.289749] LustreError: 35540:0:(osd_handler.c:720:osd_ro()) Skipped 1 previous similar message [ 1434.480874] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1439.066743] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1441.277457] Lustre: Failing over lustre-MDT0000 [ 1441.503124] Lustre: server umount lustre-MDT0000 complete [ 1445.757278] LustreError: 6392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786798872 with bad export cookie 14690694427229853791 [ 1445.759814] Lustre: Failing over lustre-MDT0001 [ 1445.763855] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.781520] LustreError: 6392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1446.253342] Lustre: server umount lustre-MDT0001 complete [ 1465.313804] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.318152] LustreError: 36749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.331562] Lustre: Skipped 15 previous similar messages [ 1465.361161] LustreError: 36749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 117 previous similar messages [ 1465.412413] LustreError: 36723:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 1465.415756] Lustre: 36723:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 1471.445247] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1471.462788] Lustre: Skipped 1 previous similar message [ 1471.992992] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 1477.959754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1313) [ 1477.965586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1313) [ 1479.621294] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1480.162119] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1490.856816] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1493.421109] Lustre: lustre-MDT0001: Denying connection for new client 682923f0-2d8a-4741-853c-fed906e6bd94 (at 192.168.202.10@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 1493.442791] Lustre: Skipped 11 previous similar messages [ 1501.151174] Lustre: 3455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786798873/real 1786798873] req@ffff96738c4c1500 x1873593000887552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786798928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1542.500235] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1542.503410] Lustre: 36803:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 912d68fc-778b-4ad7-a515-4b88ee330731@ [ 1542.511951] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1542.753699] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1542.761848] Lustre: Skipped 11 previous similar messages [ 1542.806681] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:257) [ 1542.817365] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:257) [ 1553.294593] Lustre: DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f [ 1554.773524] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 09:03:01 (1786798981) [ 1559.143927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1562.169892] LustreError: 38376:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 1562.176692] LustreError: 38376:0:(osd_handler.c:720:osd_ro()) Skipped 2 previous similar messages [ 1563.182324] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1564.951379] Lustre: Failing over lustre-MDT0000 [ 1565.159252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.232353] Lustre: server umount lustre-MDT0000 complete [ 1569.648845] LustreError: 6391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786798996 with bad export cookie 14690694427229858201 [ 1569.653954] Lustre: Failing over lustre-MDT0001 [ 1569.768470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1570.080331] Lustre: server umount lustre-MDT0001 complete [ 1590.271633] LustreError: 39379:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 1590.280515] Lustre: 39379:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 1594.336742] LustreError: 3451:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9674bc054700 x1873593000962048/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1601.836780] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:289) [ 1601.850034] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:289) [ 1603.045588] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1603.102830] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1614.844696] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1622.282688] Lustre: lustre-MDT0000: Denying connection for new client 7b923cbb-90af-4103-a2f9-bbcf02080f40 (at 192.168.202.10@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 1622.313792] Lustre: Skipped 10 previous similar messages [ 1670.500272] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1670.505364] Lustre: 39475:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 682923f0-2d8a-4741-853c-fed906e6bd94@ [ 1670.513470] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1670.590433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1345) [ 1670.595801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1345) [ 1684.541564] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 09:05:11 (1786799111) [ 1689.717295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1691.637684] Lustre: Failing over lustre-MDT0000 [ 1691.937244] Lustre: server umount lustre-MDT0000 complete [ 1709.526940] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786799120/real 1786799120] req@ffff9674bc028700 x1873593001028096/t0(0) o400->MGC192.168.202.110@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786799136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1709.554105] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1709.564186] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.577245] LustreError: Skipped 1 previous similar message [ 1710.424483] Lustre: MGS: Not available for connect from 192.168.202.10@tcp (not set up) [ 1710.434690] Lustre: Skipped 1 previous similar message [ 1721.166800] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1721.170806] Lustre: Skipped 4 previous similar messages [ 1724.686649] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1725.567073] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1725.575566] Lustre: Skipped 4 previous similar messages [ 1725.637532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1377) [ 1725.638951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1377) [ 1735.326573] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.374762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1746.920711] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 09:06:13 (1786799173) [ 1751.712781] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1754.082832] Lustre: Failing over lustre-MDT0001 [ 1754.271098] Lustre: server umount lustre-MDT0001 complete [ 1756.128842] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.155730] LustreError: Skipped 2 previous similar messages [ 1773.726802] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 1773.737818] Lustre: Skipped 4 previous similar messages [ 1777.793282] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1790.352488] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.500695] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 1848.512197] Lustre: 42870:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 7b923cbb-90af-4103-a2f9-bbcf02080f40@ [ 1848.531578] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 1848.745380] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:321) [ 1848.748578] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:321) [ 1857.293671] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 09:08:04 (1786799284) [ 1861.299943] LustreError: 43809:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1861.306676] LustreError: 43809:0:(osd_handler.c:720:osd_ro()) Skipped 2 previous similar messages [ 1862.338782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1867.425366] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 1869.523083] Lustre: Failing over lustre-MDT0000 [ 1869.912115] Lustre: server umount lustre-MDT0000 complete [ 1874.289514] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799301 with bad export cookie 14690694427229862975 [ 1874.293231] Lustre: Failing over lustre-MDT0001 [ 1874.303091] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1875.950494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1875.962710] Lustre: Skipped 1 previous similar message [ 1879.874255] Lustre: server umount lustre-MDT0001 complete [ 1900.852469] LustreError: 45017:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 1900.867279] Lustre: 45017:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 1919.328498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1919.336529] Lustre: Skipped 7 previous similar messages [ 1925.277131] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1925.733222] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1926.205991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:353) [ 1926.206856] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:353) [ 1936.975856] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1939.447170] Lustre: lustre-MDT0000: Denying connection for new client 03b4a18b-4c16-490d-b6a3-9c8459ce4512 (at 192.168.202.10@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 1939.471524] Lustre: Skipped 20 previous similar messages [ 1994.500235] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1994.508151] Lustre: 45140:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client daf0aeca-bedd-4c25-8ff2-7260e32a02cc@ [ 1994.525204] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1994.641142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1409) [ 1994.644555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1409) [ 2007.255268] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 09:10:33 (1786799433) [ 2013.320198] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2018.723567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2021.198553] Lustre: Failing over lustre-MDT0000 [ 2021.432362] Lustre: server umount lustre-MDT0000 complete [ 2023.405202] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.424767] Lustre: Skipped 21 previous similar messages [ 2025.320938] LustreError: 6390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799452 with bad export cookie 14690694427229865831 [ 2025.324897] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.325220] Lustre: Failing over lustre-MDT0001 [ 2025.334501] LustreError: 6390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.351506] LustreError: Skipped 1 previous similar message [ 2025.690990] Lustre: server umount lustre-MDT0001 complete [ 2043.871645] Lustre: 3455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786799455/real 1786799455] req@ffff967388c68a80 x1873593001201152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786799471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2046.184468] LustreError: 47616:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2046.190825] Lustre: 47616:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2054.677783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1441) [ 2054.680625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1441) [ 2056.678247] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2057.407590] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2063.588805] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2063.592455] Lustre: Skipped 22 previous similar messages [ 2068.581764] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.500218] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [ 2124.507932] Lustre: 47676:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 03b4a18b-4c16-490d-b6a3-9c8459ce4512@ [ 2124.519797] Lustre: lustre-MDT0001: disconnecting 1 stale clients [ 2124.683037] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:385) [ 2124.684940] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:385) [ 2136.902871] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 09:12:43 (1786799563) [ 2141.944145] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2146.077436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2147.995153] Lustre: Failing over lustre-MDT0000 [ 2148.128244] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (stopping) [ 2148.343267] Lustre: server umount lustre-MDT0000 complete [ 2150.890323] LustreError: 7960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.914178] LustreError: 7960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 84 previous similar messages [ 2152.752076] Lustre: Failing over lustre-MDT0001 [ 2152.754572] LustreError: 6390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799579 with bad export cookie 14690694427229868029 [ 2152.778262] LustreError: 6390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2153.024538] Lustre: server umount lustre-MDT0001 complete [ 2172.621507] LustreError: 50193:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2172.626162] Lustre: 50193:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2183.328700] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2183.412556] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:417) [ 2183.426689] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:417) [ 2184.029334] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2188.506960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1473) [ 2188.508182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1473) [ 2194.698816] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2196.561336] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2198.239247] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2207.756670] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 09:13:54 (1786799634) [ 2212.899377] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2217.370866] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2219.577684] Lustre: Failing over lustre-MDT0000 [ 2219.880608] Lustre: server umount lustre-MDT0000 complete [ 2224.414384] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799651 with bad export cookie 14690694427229870325 [ 2224.418869] Lustre: Failing over lustre-MDT0001 [ 2224.426571] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2224.749689] Lustre: server umount lustre-MDT0001 complete [ 2244.611820] LustreError: 52830:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2244.619653] Lustre: 52830:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2249.934551] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2249.950570] Lustre: Skipped 7 previous similar messages [ 2254.499805] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2254.514721] Lustre: Skipped 7 previous similar messages [ 2254.599621] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:449) [ 2254.603817] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:449) [ 2255.826306] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2256.524684] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2264.487617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1505) [ 2264.488722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1505) [ 2269.461243] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2270.899446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2272.100333] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2280.347336] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 09:15:07 (1786799707) [ 2284.378074] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2287.996442] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2290.365733] Lustre: Failing over lustre-MDT0000 [ 2290.588139] Lustre: server umount lustre-MDT0000 complete [ 2294.398862] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799721 with bad export cookie 14690694427229872600 [ 2294.410139] Lustre: Failing over lustre-MDT0001 [ 2294.416407] LustreError: 18559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2294.741768] Lustre: server umount lustre-MDT0001 complete [ 2312.732464] LustreError: 55481:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2312.735171] Lustre: 55481:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2319.846958] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcbdfd4bf36d8ead0 [ 2320.420554] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 2320.423719] Lustre: Skipped 8 previous similar messages [ 2320.654789] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 2320.664961] LustreError: Skipped 5 previous similar messages [ 2325.415110] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2325.955449] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2326.275510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1537) [ 2326.278176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1537) [ 2326.724417] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:481) [ 2326.727304] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:481) [ 2334.600763] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2336.051286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2337.666818] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2346.248189] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 09:16:12 (1786799772) [ 2347.774983] Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs [ 2349.640617] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 09:16:16 (1786799776) [ 2350.926436] Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs [ 2352.968512] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 09:16:19 (1786799779) [ 2354.557801] Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs [ 2356.310446] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 09:16:23 (1786799783) [ 2357.930653] Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs [ 2359.719334] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 09:16:26 (1786799786) [ 2361.015949] Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs [ 2362.871736] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 09:16:29 (1786799789) [ 2364.294848] Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs [ 2366.036249] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 09:16:32 (1786799792) [ 2367.372711] Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs [ 2369.032890] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 09:16:35 (1786799795) [ 2370.831725] Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs [ 2372.388799] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 09:16:39 (1786799799) [ 2374.043365] Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs [ 2375.805509] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 09:16:42 (1786799802) [ 2377.209132] Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs [ 2378.887079] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 09:16:45 (1786799805) [ 2380.432809] Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs [ 2381.809546] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 09:16:48 (1786799808) [ 2382.920745] Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs [ 2384.336564] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 09:16:51 (1786799811) [ 2385.660192] Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs [ 2387.045445] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 09:16:54 (1786799814) [ 2388.439141] Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs [ 2390.017555] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ========================================================== 09:16:57 (1786799817) [ 2392.777727] LustreError: 58272:0:(osd_handler.c:720:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** [ 2392.784379] LustreError: 58272:0:(osd_handler.c:720:osd_ro()) Skipped 9 previous similar messages [ 2393.683500] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2396.094451] Lustre: Failing over lustre-MDT0001 [ 2396.349179] Lustre: server umount lustre-MDT0001 complete [ 2420.002427] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:513) [ 2420.005155] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:513) [ 2420.792273] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2430.353298] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2432.241331] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2437.905220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2440.139774] Lustre: Failing over lustre-MDT0000 [ 2440.426298] Lustre: server umount lustre-MDT0000 complete [ 2462.491495] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2463.485729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1569) [ 2463.496152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1569) [ 2470.824357] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2472.140421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2480.567419] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ========================================================== 09:18:27 (1786799907) [ 2484.241240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2485.208706] Lustre: *** cfs_fail_loc=1702, val=0*** [ 2487.067700] Lustre: Failing over lustre-MDT0000 [ 2487.365501] Lustre: server umount lustre-MDT0000 complete [ 2509.291342] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2510.234248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1601) [ 2510.235592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1601) [ 2518.801069] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2520.623479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2528.948848] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ========================================================== 09:19:15 (1786799955) [ 2533.085895] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 2534.012082] Lustre: *** cfs_fail_loc=1702, val=0*** [ 2536.162219] Lustre: Failing over lustre-MDT0001 [ 2536.525097] Lustre: server umount lustre-MDT0001 complete [ 2554.111261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.119677] Lustre: Skipped 12 previous similar messages [ 2559.660690] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:137 to 0x2c0000400:545) [ 2559.661138] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:545) [ 2559.905432] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2570.344466] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2571.865609] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2580.950610] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 09:20:07 (1786800007) [ 2582.363275] Lustre: DEBUG MARKER: SKIP: replay-single test_117 needs >= 4 MDTs [ 2584.098681] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 09:20:11 (1786800011) [ 2585.436918] Lustre: *** cfs_fail_loc=1705, val=0*** [ 2589.770430] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2591.915158] Lustre: Failing over lustre-MDT0000 [ 2592.315206] Lustre: server umount lustre-MDT0000 complete [ 2609.570921] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.589175] LustreError: Skipped 5 previous similar messages [ 2614.417607] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2615.644344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1633) [ 2615.645315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1633) [ 2623.029867] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.674485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2631.833487] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 09:20:59 (1786800059) [ 2636.125616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2639.084661] Lustre: Failing over lustre-MDT0000 [ 2639.659547] Lustre: server umount lustre-MDT0000 complete [ 2640.871370] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.887798] Lustre: Skipped 41 previous similar messages [ 2652.428253] Lustre: 56078:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2653.635842] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2654.178519] Lustre: 7960:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2654.210764] LustreError: 66766:0:(ldlm_lib.c:2689:replay_request_or_update()) cfs_fail_timeout id 714 sleeping for 65000ms [ 2661.017839] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2719.287225] LustreError: 66766:0:(ldlm_lib.c:2689:replay_request_or_update()) cfs_fail_timeout id 714 awake [ 2719.293227] Lustre: 66766:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp [ 2719.317548] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2719.327743] Lustre: 66766:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@ffff9674891a8a80 x1873592978844416/t0(85899345926) o36->557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp:550/0 lens 552/0 e 7 to 0 dl 1786800160 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'mkdir.0' uid:0 gid:0 projid:4294967295 [ 2719.379651] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2719.391937] Lustre: 66766:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2719.439943] Lustre: lustre-MDT0000: Denying connection for new client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 1 evicted) already passed deadline 0:06 [ 2719.469173] Lustre: Skipped 21 previous similar messages [ 2719.764795] Lustre: 66766:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2719.779840] Lustre: 66766:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2719.792799] Lustre: 66766:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2720.079383] Lustre: lustre-MDT0000-osd: cancel update llog [0x200002340:0x1:0x0] [ 2720.237752] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240002342:0x1:0x0] [ 2720.310679] Lustre: 66766:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 2720.323652] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2720.335271] Lustre: Skipped 42 previous similar messages [ 2720.343056] LustreError: dumping log to /tmp/lustre-log.1786800147.66766 [ 2720.578641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1665) [ 2720.583036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1665) [ 2727.222348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 58 sec [ 2740.992456] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 09:22:47 (1786800167) [ 2743.747104] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2751.410446] Lustre: Failing over lustre-MDT0000 [ 2751.766461] Lustre: server umount lustre-MDT0000 complete [ 2754.864893] LustreError: 55483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.10@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.879726] LustreError: 55483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 119 previous similar messages [ 2760.895483] Lustre: lustre-MDT0000: Aborting client recovery [ 2760.899330] LustreError: 68334:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2760.903275] Lustre: 68368:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2760.906087] Lustre: 68368:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 1 previous similar message [ 2760.954298] Lustre: lustre-MDT0000-osd: cancel update llog [0x20000a040:0x3:0x0] [ 2761.021914] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400088d2:0x1:0x0] [ 2761.097603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1201 to 0x2c0000401:1697) [ 2761.101431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1697) [ 2765.009903] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2765.802379] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2786.161629] Lustre: DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 09:23:32 (1786800212) [ 2789.768585] Lustre: Failing over lustre-MDT0000 [ 2790.191656] Lustre: server umount lustre-MDT0000 complete [ 2795.490693] Lustre: *** cfs_fail_loc=721, val=0*** [ 2795.500076] Lustre: Skipped 2 previous similar messages [ 2796.513219] Lustre: *** cfs_fail_loc=721, val=0*** [ 2796.514817] Lustre: Skipped 9 previous similar messages [ 2800.607805] Lustre: *** cfs_fail_loc=721, val=0*** [ 2800.905152] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.10@tcp (not set up) [ 2805.727640] Lustre: *** cfs_fail_loc=721, val=1*** [ 2805.729489] Lustre: Skipped 105 previous similar messages [ 2806.295916] Lustre: *** cfs_fail_loc=721, val=1*** [ 2806.305437] Lustre: Skipped 6 previous similar messages [ 2806.422967] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2810.848144] Lustre: *** cfs_fail_loc=721, val=1*** [ 2810.854237] Lustre: Skipped 23 previous similar messages [ 2817.873164] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 2821.088949] Lustre: *** cfs_fail_loc=721, val=1*** [ 2821.096208] Lustre: Skipped 35 previous similar messages [ 2834.201382] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [ 2836.448463] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786800233/real 1786800233] req@ffff9674b7ff2300 x1873593001784960/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1786800263 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2836.487494] Lustre: 3451:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 46 previous similar messages [ 2836.503978] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2836.534180] Lustre: *** cfs_fail_loc=721, val=1*** [ 2836.536885] Lustre: 69769:0:(tgt_handler.c:581:tgt_handle_recovery()) @@@ obsoleted, rq_xid=1873592978993664, exp_last_xid=1873592978995455 req@ffff96738c594e00 x1873592978993664/t0(0) o101->557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp:0/0 lens 328/0 e 0 to 0 dl 1786800240 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2837.766266] Lustre: *** cfs_fail_loc=721, val=1*** [ 2837.768416] Lustre: Skipped 63 previous similar messages [ 2850.593393] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [ 2865.932083] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:06 [ 2866.657420] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2866.669368] Lustre: *** cfs_fail_loc=721, val=1*** [ 2866.672639] Lustre: 69769:0:(tgt_handler.c:581:tgt_handle_recovery()) @@@ obsoleted, rq_xid=1873592978993792, exp_last_xid=1873592978995455 req@ffff96738c594000 x1873592978993792/t0(0) o101->557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp:0/0 lens 328/0 e 0 to 0 dl 1786800240 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2870.473908] Lustre: *** cfs_fail_loc=721, val=1*** [ 2870.477597] Lustre: Skipped 115 previous similar messages [ 2882.342441] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:05 [ 2896.863897] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2896.884509] Lustre: *** cfs_fail_loc=721, val=1*** [ 2896.890802] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2897.681337] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:29 [ 2914.080203] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:13 [ 2927.076599] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2927.093577] Lustre: *** cfs_fail_loc=721, val=1*** [ 2936.076247] Lustre: *** cfs_fail_loc=721, val=1*** [ 2936.080783] Lustre: Skipped 253 previous similar messages [ 2957.279778] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2957.291384] Lustre: *** cfs_fail_loc=721, val=1*** [ 2957.303456] Lustre: 69769:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2957.314381] Lustre: 69769:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 20 previous similar messages [ 2962.205315] Lustre: lustre-MDT0000: Client 557978e1-20e5-4032-a134-521998a7327a (at 192.168.202.10@tcp) reconnected, waiting for 2 clients in recovery for 0:20 [ 2962.225856] Lustre: Skipped 2 previous similar messages [ 2987.489500] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 2987.496341] Lustre: 69769:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2987.505769] Lustre: 69769:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2987.512137] Lustre: 69769:0:(ldlm_lib.c:2393:target_recovery_overseer()) Skipped 1 previous similar message [ 2987.517462] Lustre: 69769:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2987.530143] Lustre: 69769:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2987.536396] Lustre: 69769:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp [ 2987.547400] Lustre: 69769:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 2 previous similar messages [ 2987.551989] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2987.555502] Lustre: Skipped 1 previous similar message [ 2987.558736] LustreError: 69769:0:(ldlm_lib.c:1933:abort_lock_replay_queue()) @@@ aborted: req@ffff9674bbfb6300 x1873592979000192/t0(0) o101->557978e1-20e5-4032-a134-521998a7327a@192.168.202.10@tcp:0/0 lens 328/0 e 0 to 0 dl 1786800288 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2987.623428] Lustre: lustre-MDT0000-osd: cancel update llog [0x20000a810:0x1:0x0] [ 2987.701059] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400088d3:0x1:0x0] [ 2987.739345] Lustre: 69769:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 2987.751344] LustreError: dumping log to /tmp/lustre-log.1786800414.69769 [ 2987.823531] Lustre: lustre-MDT0000: Recovery over after 3:05, of 2 clients 1 recovered and 1 was evicted. [ 2987.828858] Lustre: Skipped 9 previous similar messages [ 2987.859052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1699 to 0x2c0000401:1729) [ 2987.860858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1729) [ 3004.499271] Lustre: DEBUG MARKER: == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 09:27:11 (1786800431) [ 3007.464935] LustreError: 70637:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3007.469697] LustreError: 70637:0:(osd_handler.c:720:osd_ro()) Skipped 6 previous similar messages [ 3008.364453] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3010.613548] Lustre: Failing over lustre-MDT0000 [ 3011.000915] Lustre: server umount lustre-MDT0000 complete [ 3029.331899] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3029.336972] Lustre: Skipped 11 previous similar messages [ 3029.875669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3029.890051] Lustre: Skipped 10 previous similar messages [ 3033.952332] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3034.822212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1761) [ 3034.823117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1699 to 0x2c0000401:1761) [ 3042.541553] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3044.019185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3053.508382] Lustre: DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 09:28:00 (1786800480) [ 3057.068324] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3059.435150] Lustre: Failing over lustre-MDT0000 [ 3059.785544] Lustre: server umount lustre-MDT0000 complete [ 3092.789468] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3092.842915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1793) [ 3092.843717] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1699 to 0x2c0000401:1793) [ 3102.415571] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3103.962199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3112.728961] Lustre: DEBUG MARKER: == replay-single test 131a: DoM file write lock replay === 09:28:59 (1786800539) [ 3117.389402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3119.664903] Lustre: Failing over lustre-MDT0000 [ 3119.967766] Lustre: server umount lustre-MDT0000 complete [ 3138.529085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3143.627126] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3144.469301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1201 to 0x280000401:1825) [ 3144.480514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1699 to 0x2c0000401:1825) [ 3153.838210] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3155.651710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3164.924676] Lustre: DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b [ 3166.762508] Lustre: DEBUG MARKER: == replay-single test 132a: PFL new component instantiate replay ========================================================== 09:29:53 (1786800593) [ 3170.502928] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3172.824747] Lustre: Failing over lustre-MDT0000 [ 3173.179768] Lustre: server umount lustre-MDT0000 complete [ 3191.667272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.671580] Lustre: Skipped 7 previous similar messages [ 3196.548392] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3197.099934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1857) [ 3197.101244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1857) [ 3205.108740] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3206.314843] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3215.571797] Lustre: DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 09:30:42 (1786800642) [ 3221.087767] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3221.089659] Lustre: Skipped 264 previous similar messages [ 3224.007311] Lustre: Failing over lustre-MDT0000 [ 3224.315984] Lustre: server umount lustre-MDT0000 complete [ 3237.654034] Lustre: lustre-MDT0001: Client 14651699-6507-4a5f-a649-7a6a3ee6afa1 (at 192.168.202.10@tcp) reconnecting [ 3242.419767] LustreError: MGC192.168.202.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3242.427875] LustreError: Skipped 7 previous similar messages [ 3248.072868] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3248.397270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:mdt [ 3248.423677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:mdt] [ 3248.470722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1889) [ 3248.475621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1889) [ 3258.309775] Lustre: DEBUG MARKER: == replay-single test 134: replay creation of a file created in a pool ========================================================== 09:31:25 (1786800685) [ 3270.749969] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3272.715510] Lustre: Failing over lustre-MDT0000 [ 3272.929580] Lustre: server umount lustre-MDT0000 complete [ 3273.706425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.721684] Lustre: Skipped 32 previous similar messages [ 3304.580658] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3305.029126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1921) [ 3305.030769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1827 to 0x280000401:1921) [ 3314.178633] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3315.891720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3335.135663] Lustre: DEBUG MARKER: == replay-single test 135: Server failure in lock replay phase ========================================================== 09:32:42 (1786800762) [ 3341.526477] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3343.502083] Lustre: Failing over lustre-OST0000 [ 3343.732774] Lustre: server umount lustre-OST0000 complete [ 3350.438190] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 3356.132992] LustreError: 23888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.149641] LustreError: 23888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 156 previous similar messages [ 3358.729322] Lustre: *** cfs_fail_loc=32d, val=20*** [ 3358.735645] Lustre: Skipped 1 previous similar message [ 3362.934310] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3369.807758] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3371.256604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec [ 3373.144745] Lustre: Failing over lustre-OST0000 [ 3373.157073] LustreError: 81760:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 3373.162433] Lustre: 80977:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3373.167824] LustreError: 80977:0:(ofd_obd.c:1324:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [ 3373.360124] Lustre: server umount lustre-OST0000 complete [ 3390.775845] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 3397.952109] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3397.960776] Lustre: Skipped 32 previous similar messages [ 3403.089766] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3407.088129] LustreError: 82999:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x785:0x0].0x0 (ffff96748a87d600) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3408.352026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.353230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.364898] LustreError: Skipped 4 previous similar messages [ 3408.385028] Lustre: Skipped 4 previous similar messages [ 3409.299140] Lustre: server umount lustre-OST0000 complete [ 3415.818449] Lustre: server umount lustre-OST0001 complete [ 3431.024172] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3436.523940] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3436.550319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3437.017461] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786800848/real 1786800848] req@ffff9674bc20ca80 x1873593002145920/t0(0) o400->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786800864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3437.052196] Lustre: 3453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 3441.251247] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3441.277686] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3445.433809] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3456.691891] Lustre: DEBUG MARKER: == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 09:34:43 (1786800883) [ 3458.533949] Lustre: DEBUG MARKER: SKIP: replay-single test_136 needs > 2 MDTs [ 3460.293821] Lustre: DEBUG MARKER: == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 09:34:47 (1786800887) [ 3464.720567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3466.448044] Lustre: Failing over lustre-MDT0000 [ 3466.708266] Lustre: server umount lustre-MDT0000 complete [ 3488.307911] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3488.957161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1943 to 0x280000401:1985) [ 3488.971295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1953) [ 3498.122533] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3499.776950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3510.508597] Lustre: DEBUG MARKER: == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 09:35:36 (1786800936) [ 3514.912962] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3517.436960] Lustre: Failing over lustre-MDT0001 [ 3517.781681] Lustre: server umount lustre-MDT0001 complete [ 3541.197088] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3542.233832] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:549 to 0x2c0000400:577) [ 3542.267375] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:577) [ 3551.365698] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3553.430288] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3563.972979] Lustre: DEBUG MARKER: == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 09:36:30 (1786800990) [ 3568.906463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3572.875529] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3574.744760] Lustre: Failing over lustre-MDT0001 [ 3575.010346] Lustre: server umount lustre-MDT0001 complete [ 3579.114894] Lustre: Failing over lustre-MDT0000 [ 3579.429559] Lustre: server umount lustre-MDT0000 complete [ 3598.450640] LustreError: 89568:0:(llog.c:1655:llog_backup()) MGC192.168.202.110@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3598.457216] LustreError: 89580:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.110@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9674814b0d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3598.467458] Lustre: 89568:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.110@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3603.408649] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3603.928891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3604.810675] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3604.819889] Lustre: Skipped 10 previous similar messages [ 3604.884316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1943 to 0x280000401:2017) [ 3604.888041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:1985) [ 3605.217950] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:549 to 0x2c0000400:609) [ 3605.218322] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:609) [ 3613.528259] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3614.900058] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3616.313609] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3625.704916] Lustre: DEBUG MARKER: == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 09:37:32 (1786801052) [ 3627.719698] Lustre: DEBUG MARKER: SKIP: replay-single test_200 Need remote client [ 3629.752863] Lustre: DEBUG MARKER: == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 09:37:36 (1786801056) [ 3634.504090] LustreError: 89740:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [ 3638.535827] LustreError: 83613:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [ 3638.549419] LustreError: 83613:0:(tgt_handler.c:1143:tgt_disconnect()) Skipped 1 previous similar message [ 3642.527232] LustreError: 89740:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 awake [ 3642.544361] Lustre: Failing over lustre-MDT0001 [ 3642.575242] LustreError: 83624:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [ 3642.610080] LustreError: 83624:0:(tgt_handler.c:1143:tgt_disconnect()) Skipped 2 previous similar messages [ 3646.567567] LustreError: 83613:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 awake [ 3646.576983] LustreError: 83613:0:(tgt_handler.c:1143:tgt_disconnect()) Skipped 1 previous similar message [ 3650.615143] LustreError: 83615:0:(tgt_handler.c:1143:tgt_disconnect()) cfs_fail_timeout id 245 awake [ 3650.824171] Lustre: server umount lustre-MDT0001 complete [ 3659.741395] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3659.746035] Lustre: Skipped 11 previous similar messages [ 3660.911284] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3660.924630] Lustre: Skipped 11 previous similar messages [ 3665.056155] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:549 to 0x2c0000400:641) [ 3665.057078] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:137 to 0x280000400:641) [ 3665.119237] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3675.570081] Lustre: DEBUG MARKER: == replay-single test 202: pfl replay should recovery layout generation ========================================================== 09:38:21 (1786801101) [ 3679.793608] LustreError: 92357:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3679.800953] LustreError: 92357:0:(osd_handler.c:720:osd_ro()) Skipped 9 previous similar messages [ 3680.663974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3682.431476] Lustre: Failing over lustre-MDT0000 [ 3682.662312] Lustre: server umount lustre-MDT0000 complete [ 3706.696775] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3706.705863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2019 to 0x280000401:2049) [ 3706.705941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1828 to 0x2c0000401:2017) [ 3716.754376] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3718.701424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3728.349732] Lustre: DEBUG MARKER: == replay-single test 203: resend can hit original request ========================================================== 09:39:14 (1786801154) [ 3730.248410] LustreError: 91159:0:(mdt_handler.c:2126:mdt_getattr_name_lock()) cfs_fail_timeout id 2403 sleeping for 2000ms [ 3732.343113] LustreError: 91159:0:(mdt_handler.c:2126:mdt_getattr_name_lock()) cfs_fail_timeout id 2403 awake [ 3732.355847] LustreError: 91159:0:(mdt_handler.c:2126:mdt_getattr_name_lock()) Skipped 2 previous similar messages [ 3732.366428] Lustre: 91159:0:(service.c:2628:ptlrpc_server_handle_request()) @@@ pause req after reply req@ffff96738290c000 x1873592979293568/t0(0) o101->14651699-6507-4a5f-a649-7a6a3ee6afa1@192.168.202.10@tcp:87/0 lens 592/1888 e 0 to 0 dl 1786801207 ref 1 fl Complete:/600/0 rc 0/0 job:'stat.0' uid:0 gid:0 projid:0 [ 3735.455313] Lustre: 91159:0:(service.c:2630:ptlrpc_server_handle_request()) @@@ continue req@ffff96738290c000 x1873592979293568/t0(0) o101->14651699-6507-4a5f-a649-7a6a3ee6afa1@192.168.202.10@tcp:87/0 lens 592/1888 e 0 to 0 dl 1786801207 ref 1 fl Complete:/600/0 rc 0/0 job:'stat.0' uid:0 gid:0 projid:0 [ 3743.638251] Lustre: DEBUG MARKER: == replay-single test complete, duration 3447 sec ======== 09:39:30 (1786801170) [ 3745.461517] Lustre: DEBUG MARKER: === replay-single: start cleanup 09:39:32 (1786801172) === [ 3752.592694] Lustre: DEBUG MARKER: === replay-single: finish cleanup 09:39:39 (1786801179) === [ 3754.894436] Lustre: Failing over lustre-MDT0000 [ 3755.102145] Lustre: server umount lustre-MDT0000 complete [ 3783.151455] LustreError: 3451:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9674b7977100 x1873593002424192/t0(0) o250->MGC192.168.202.110@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3788.652700] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3789.161629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2051 to 0x280000401:2081) [ 3789.170907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2019 to 0x2c0000401:2049) [ 3797.493366] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3800.103441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3813.671304] Lustre: server umount lustre-MDT0000 complete [ 3818.250440] LustreError: 6391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801245 with bad export cookie 14690694427229930924 [ 3818.261909] LustreError: 6391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3818.784250] Lustre: server umount lustre-MDT0001 complete [ 3833.342301] Lustre: server umount lustre-OST0000 complete [ 3847.820855] Lustre: server umount lustre-OST0001 complete [ 3860.143891] Lustre: DEBUG MARKER: oleg210-server.virtnet: executing unload_modules_local [ 3863.239194] Key type lgssc unregistered [ 3863.537692] LNet: 96667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.554579] LNetError: 96667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.588114] LNet: Removed LNI 192.168.202.110@tcp [ 3864.854418] Key type .llcrypt unregistered [ 3864.859915] Key type ._llcrypt unregistered