[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469569581 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003129] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008402] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012147] LSM: Security Framework initializing [ 0.013067] Yama: becoming mindful. [ 0.014033] SELinux: Initializing. [ 0.015046] *** VALIDATE selinux *** [ 0.022127] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027359] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] *** VALIDATE tmpfs *** [ 0.031451] *** VALIDATE proc *** [ 0.032256] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.040526] debug: unmapping init [mem 0xffffffff8b259000-0xffffffff8b260fff] [ 0.042961] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.044886] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049008] ... event mask: 000000070000000f [ 0.050213] rcu: Hierarchical SRCU implementation. [ 0.052318] smp: Bringing up secondary CPUs ... [ 0.053530] x86: Booting SMP configuration: [ 0.054031] .... node #0, CPUs: #1 #2 #3 [ 0.057456] smp: Brought up 1 node, 4 CPUs [ 0.059011] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131647] node 0 deferred pages initialised in 68ms [ 0.134027] devtmpfs: initialized [ 0.135488] x86/mm: Memory block size: 128MB [ 0.139617] gcov: version magic: 0x41383552 [ 0.142227] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144220] pinctrl core: initialized pinctrl subsystem [ 0.146281] [ 0.146788] ************************************************************* [ 0.149012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ** ** [ 0.152012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155013] ** ** [ 0.157010] ** This means that this kernel is built to expose internal ** [ 0.158009] ** IOMMU data structures, which may compromise security on ** [ 0.160011] ** your system. ** [ 0.162011] ** ** [ 0.164016] ** If you see this message and you are not debugging the ** [ 0.166012] ** kernel, report this immediately to your vendor! ** [ 0.168011] ** ** [ 0.170009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172010] ************************************************************* [ 0.174658] NET: Registered protocol family 16 [ 0.176464] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183173] cpuidle: using governor menu [ 0.185128] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186406] PCI: Using configuration type 1 for base access [ 0.187178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197033] cryptd: max_cpu_qlen set to 1000 [ 0.199235] ACPI: Added _OSI(Module Device) [ 0.201011] ACPI: Added _OSI(Processor Device) [ 0.202009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207060] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212312] ACPI: Interpreter enabled [ 0.214061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215010] ACPI: Using IOAPIC for interrupt routing [ 0.217185] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231578] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244865] acpiphp: Slot [2] registered [ 0.246084] acpiphp: Slot [5] registered [ 0.247036] acpiphp: Slot [6] registered [ 0.248064] acpiphp: Slot [7] registered [ 0.249116] acpiphp: Slot [8] registered [ 0.251133] acpiphp: Slot [9] registered [ 0.252111] acpiphp: Slot [10] registered [ 0.254131] acpiphp: Slot [3] registered [ 0.255101] acpiphp: Slot [4] registered [ 0.256083] acpiphp: Slot [11] registered [ 0.257070] acpiphp: Slot [12] registered [ 0.258079] acpiphp: Slot [13] registered [ 0.260087] acpiphp: Slot [14] registered [ 0.261074] acpiphp: Slot [15] registered [ 0.262089] acpiphp: Slot [16] registered [ 0.263062] acpiphp: Slot [17] registered [ 0.264064] acpiphp: Slot [18] registered [ 0.265103] acpiphp: Slot [19] registered [ 0.267074] acpiphp: Slot [20] registered [ 0.268026] acpiphp: Slot [21] registered [ 0.269081] acpiphp: Slot [22] registered [ 0.269936] acpiphp: Slot [23] registered [ 0.270052] acpiphp: Slot [24] registered [ 0.270926] acpiphp: Slot [25] registered [ 0.272063] acpiphp: Slot [26] registered [ 0.272980] acpiphp: Slot [27] registered [ 0.274065] acpiphp: Slot [28] registered [ 0.274964] acpiphp: Slot [29] registered [ 0.276047] acpiphp: Slot [30] registered [ 0.277075] acpiphp: Slot [31] registered [ 0.278053] PCI host bridge to bus 0000:00 [ 0.279012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.283020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.285026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.289017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.291163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293928] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297287] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.306651] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.311051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.317013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.319452] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323104] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329784] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.354902] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360862] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394641] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.432013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440285] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.447013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.452015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.467016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.475779] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.482013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.486018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.500019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.508222] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.516014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.523015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.538016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.547169] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.553013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.561015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.579015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.590728] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.592207] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.594349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.597362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.600202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.604098] iommu: Default domain type: Passthrough [ 0.605309] SCSI subsystem initialized [ 0.607114] ACPI: bus type USB registered [ 0.607902] usbcore: registered new interface driver usbfs [ 0.609069] usbcore: registered new interface driver hub [ 0.610059] usbcore: registered new device driver usb [ 0.612150] pps_core: LinuxPPS API ver. 1 registered [ 0.613007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.615044] PTP clock support registered [ 0.617128] EDAC MC: Ver: 3.0.0 [ 0.619130] PCI: Using ACPI for IRQ routing [ 0.620773] NetLabel: Initializing [ 0.622012] NetLabel: domain hash size = 128 [ 0.623012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626084] NetLabel: unlabeled traffic allowed by default [ 0.628131] vgaarb: loaded [ 0.629258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.631015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.638037] clocksource: Switched to clocksource kvm-clock [ 0.743101] VFS: Disk quotas dquot_6.6.0 [ 0.744563] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.746782] *** VALIDATE ramfs *** [ 0.747681] *** VALIDATE hugetlbfs *** [ 0.749029] pnp: PnP ACPI init [ 0.751301] pnp: PnP ACPI: found 6 devices [ 0.766852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.770034] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772306] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.774394] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.776128] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.777925] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.780397] NET: Registered protocol family 2 [ 0.782558] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.787226] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.791662] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.797552] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.801316] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804604] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807481] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.809747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812164] NET: Registered protocol family 1 [ 0.814269] RPC: Registered named UNIX socket transport module. [ 0.816146] RPC: Registered udp transport module. [ 0.817452] RPC: Registered tcp transport module. [ 0.818982] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.820845] NET: Registered protocol family 44 [ 0.822169] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.823834] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.825529] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.827402] PCI: CLS 0 bytes, default 64 [ 0.829691] Unpacking initramfs... [ 2.227621] debug: unmapping init [mem 0xffff9ec2bcc54000-0xffff9ec2bffbffff] [ 2.231797] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.234337] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.237726] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.737630] Initialise system trusted keyrings [ 2.739249] Key type blacklist registered [ 2.741173] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.750713] zbud: loaded [ 2.753815] *** VALIDATE nfs *** [ 2.755157] *** VALIDATE nfs4 *** [ 2.756688] pstore: using deflate compression [ 2.759935] Platform Keyring initialized [ 2.856413] NET: Registered protocol family 38 [ 2.858471] Key type asymmetric registered [ 2.859720] Asymmetric key parser 'x509' registered [ 2.861949] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.864700] io scheduler mq-deadline registered [ 2.865806] io scheduler kyber registered [ 2.867234] io scheduler bfq registered [ 2.869099] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.871637] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.873725] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.876426] ACPI: Power Button [PWRF] [ 2.882285] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.891724] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.906554] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.912900] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.931487] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.962153] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.993925] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.998511] Non-volatile memory driver v1.3 [ 2.999894] Linux agpgart interface v0.103 [ 3.026221] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.028229] vda: detected capacity change from 0 to 76779520 [ 3.041088] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.043596] vdb: detected capacity change from 0 to 1073741824 [ 3.059384] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.062594] vdc: detected capacity change from 0 to 2621440000 [ 3.077379] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.080539] vdd: detected capacity change from 0 to 2621440000 [ 3.095888] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.099031] vde: detected capacity change from 0 to 4294967296 [ 3.113385] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.115984] vdf: detected capacity change from 0 to 4294967296 [ 3.122815] libphy: Fixed MDIO Bus: probed [ 3.128576] usbcore: registered new interface driver usbserial_generic [ 3.131387] usbserial: USB Serial support registered for generic [ 3.133848] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.138751] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.140947] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.143643] mousedev: PS/2 mouse device common for all mice [ 3.146711] rtc_cmos 00:05: RTC can wake from S4 [ 3.150734] rtc_cmos 00:05: registered as rtc0 [ 3.154153] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155089] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.161405] intel_pstate: CPU model not supported [ 3.164413] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.165817] hid: raw HID events driver (C) Jiri Kosina [ 3.170288] usbcore: registered new interface driver usbhid [ 3.172288] usbhid: USB HID core driver [ 3.172534] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.173678] drop_monitor: Initializing network drop monitor service [ 3.178274] Initializing XFRM netlink socket [ 3.179975] NET: Registered protocol family 10 [ 3.182221] Segment Routing with IPv6 [ 3.183435] NET: Registered protocol family 17 [ 3.185552] mpls_gso: MPLS GSO support [ 3.190127] RAS: Correctable Errors collector initialized. [ 3.191811] AVX version of gcm_enc/dec engaged. [ 3.193428] AES CTR mode by8 optimization enabled [ 3.269049] sched_clock: Marking stable (3269022866, 0)->(4153159500, -884136634) [ 3.272671] registered taskstats version 1 [ 3.274467] Loading compiled-in X.509 certificates [ 3.276750] zswap: loaded using pool lzo/zbud [ 3.297811] Key type big_key registered [ 3.307609] Key type encrypted registered [ 3.308824] ima: No TPM chip found, activating TPM-bypass! [ 3.310380] ima: Allocated hash algorithm: sha1 [ 3.320477] ima: No architecture policies found [ 3.322784] evm: Initialising EVM extended attributes: [ 3.325136] evm: security.selinux [ 3.327124] evm: security.ima [ 3.329010] evm: security.capability [ 3.330383] evm: HMAC attrs: 0x1 [ 3.333654] rtc_cmos 00:05: setting system clock to 2026-09-14 16:21:21 UTC (1789402881) [ 3.340525] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.344304] debug: unmapping init [mem 0xffffffff8af82000-0xffffffff8b258fff] [ 3.356261] Write protecting the kernel read-only data: 28672k [ 3.360120] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 3.363414] debug: unmapping init [mem 0xffffffff89f14000-0xffffffff89ffffff] [ 3.407192] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.419900] systemd[1]: Detected virtualization kvm. [ 3.424224] systemd[1]: Detected architecture x86-64. [ 3.426365] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.455962] systemd[1]: No hostname configured. [ 3.458288] systemd[1]: Set hostname to . [ 3.461135] random: systemd: uninitialized urandom read (16 bytes read) [ 3.464550] systemd[1]: Initializing machine ID from random generator. [ 3.587240] random: systemd: uninitialized urandom read (16 bytes read) [ 3.588745] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.592697] random: systemd: uninitialized urandom read (16 bytes read) [ 3.594824] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.598721] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.238543] device-mapper: uevent: version 1.0.3 [ 4.240907] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.941959] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.965236] random: fast init done [ 5.018751] scsi host0: ata_piix [ 5.046340] scsi host1: ata_piix [ 5.048369] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.051052] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.868874] random: crng init done [ 9.876902] random: 7 urandom warning(s) missed due to ratelimiting [ 11.533503] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.370073] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.454324] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.363428] SELinux: Disabled at runtime. [ 17.511476] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.531621] systemd[1]: Detected virtualization kvm. [ 17.534963] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.988489] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.993300] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.005414] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.013694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.017736] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.031134] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.047201] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice[ 19.132144] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.356600] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.280618] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.314642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.728422] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.785074] EDAC sbridge: Ver: 1.1.2 [ 26.153952] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 27.029346] NFS: Registering the id_resolver key type [ 27.034403] Key type id_resolver registered [ 27.037300] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg307-server login: [ 66.311943] hrtimer: interrupt took 11062824 ns [ 76.238696] spl: loading out-of-tree module taints kernel. [ 83.335662] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.844855] Key type ._llcrypt registered [ 97.847220] Key type .llcrypt registered [ 97.983698] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 116.790735] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 118.434699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 118.462820] alg: No test for adler32 (adler32-zlib) [ 119.894545] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 120.940819] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 122.632194] Key type lgssc registered [ 124.220782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.906617] vdc: vdc1 vdc9 [ 143.466648] vde: vde1 vde9 [ 152.964775] vdf: vdf1 vdf9 [ 173.618883] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 184.083936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.528803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.776128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.880193] Lustre: lustre-MDT0000: new disk, initializing [ 186.380836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.464473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.035554] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.820788] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.278095] Lustre: lustre-OST0000: new disk, initializing [ 203.284550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.288866] Lustre: Skipped 1 previous similar message [ 203.372981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.415772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 208.422172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 208.513567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 209.211169] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.993871] Lustre: lustre-OST0001: new disk, initializing [ 219.997257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.173744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.461842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 221.480359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 221.684863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 228.464930] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.449655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.948943] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.470842] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.446955] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing yml_node [ 267.754971] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 270.344535] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 272.522558] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 274.127554] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-dual ============----- Mon Sep 14 12:25:51 EDT 2026 [ 290.681124] Lustre: DEBUG MARKER: excepting tests: 14b 21b 21b [ 292.311154] Lustre: DEBUG MARKER: skipping tests SLOW=no: 21b [ 294.082302] Lustre: DEBUG MARKER: === replay-dual: start setup 12:26:10 (1789403170) === [ 299.980982] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing check_config_client /mnt/lustre [ 316.235800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 319.535293] Lustre: 11359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 323.740281] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 327.562642] Lustre: DEBUG MARKER: === replay-dual: finish setup 12:26:44 (1789403204) === [ 329.507505] Lustre: DEBUG MARKER: == replay-dual test 0a: expired recovery with lost client ========================================================== 12:26:46 (1789403206) [ 329.643861] LustreError: 7269:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 329.648772] LustreError: 7269:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 333.399066] LustreError: 11857:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 334.272788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 334.439844] LustreError: 7269:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 334.455571] LustreError: 7269:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 335.465508] LustreError: 5841:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 335.483467] LustreError: 5841:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 15 previous similar messages [ 335.494231] LustreError: 5841:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 335.499627] LustreError: 5841:0:(mdt_open.c:646:mdt_finish_open()) Skipped 15 previous similar messages [ 338.671590] Lustre: Failing over lustre-MDT0000 [ 338.932832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.935443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.946635] Lustre: Skipped 1 previous similar message [ 338.954511] Lustre: Skipped 1 previous similar message [ 339.133265] Lustre: server umount lustre-MDT0000 complete [ 356.730279] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.012484] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (not set up) [ 357.214842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.394122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 358.578154] LustreError: 12505:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 358.585825] LustreError: 12505:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 34 previous similar messages [ 358.612832] LustreError: 12505:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 358.624265] LustreError: 12505:0:(mdt_open.c:646:mdt_finish_open()) Skipped 34 previous similar messages [ 359.227475] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 362.107609] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 461.500236] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 461.516083] Lustre: 12505:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client bb494820-e947-4747-a8bb-7380d63bf572@192.168.203.7@tcp [ 461.534157] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 461.626669] Lustre: 12505:0:(ldlm_lib.c:2985:target_recovery_thread()) too long recovery - read logs [ 461.637584] LustreError: dumping log to /tmp/lustre-log.1789403339.12505 [ 461.730945] Lustre: lustre-MDT0000: Recovery over after 1:43, of 2 clients 1 recovered and 1 was evicted. [ 461.766883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:28 to 0x280000400:65) [ 461.767141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:28 to 0x240000400:65) [ 482.953990] Lustre: DEBUG MARKER: == replay-dual test 0b: lost client during waiting for next transno ========================================================== 12:29:20 (1789403360) [ 485.989783] LustreError: 13256:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 486.789395] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 486.899953] LustreError: 12882:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 486.913133] LustreError: 12882:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 50 previous similar messages [ 486.922791] LustreError: 12882:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 486.935646] LustreError: 12882:0:(mdt_open.c:646:mdt_finish_open()) Skipped 50 previous similar messages [ 489.185935] Lustre: Failing over lustre-MDT0000 [ 489.615320] Lustre: server umount lustre-MDT0000 complete [ 506.636477] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.899616] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.913310] Lustre: Skipped 1 previous similar message [ 507.029543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.937462] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403370/real 1789403370] req@ffff9ec332630e00 x1876325039173504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 507.964942] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 510.730551] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 512.490793] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 512.500718] Lustre: Skipped 1 previous similar message [ 513.507817] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403375/real 1789403375] req@ffff9ec2031b2d80 x1876325039173888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 518.624849] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403380/real 1789403380] req@ffff9ec33fdc2a00 x1876325039174144/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 518.651924] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 523.300395] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 523.312507] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 528.665628] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 533.790129] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 538.902874] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 544.019284] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 554.258319] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 554.275960] Lustre: Skipped 1 previous similar message [ 574.746404] Lustre: lustre-MDT0000: Denying connection for new client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:08 [ 574.761087] Lustre: Skipped 3 previous similar messages [ 583.500166] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 583.508571] Lustre: 13893:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d16ef54d-7fdb-497e-bcf6-1b5bb89d6c83@ [ 583.536134] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 583.621143] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 0 recovered and 2 were evicted. [ 583.672113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:28 to 0x280000400:97) [ 583.679521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:28 to 0x240000400:97) [ 592.723936] Lustre: DEBUG MARKER: == replay-dual test 1: |X| simple create ================= 12:31:09 (1789403469) [ 592.811487] LustreError: 14283:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 592.817628] LustreError: 14283:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 592.824596] LustreError: 14283:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 592.831346] LustreError: 14283:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 595.969426] LustreError: 14646:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 596.938974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 599.040082] Lustre: Failing over lustre-MDT0000 [ 599.331754] Lustre: server umount lustre-MDT0000 complete [ 614.881208] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403477/real 1789403477] req@ffff9ec32d76fb80 x1876325039197312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.927941] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.683405] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.482956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.553767] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.320123] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403482/real 1789403482] req@ffff9ec203ad2a00 x1876325039197696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.350832] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 623.047261] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 625.663434] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 625.670821] Lustre: Skipped 1 previous similar message [ 626.974353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 627.050702] LustreError: 15279:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 627.059353] LustreError: 15279:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 627.063858] LustreError: 15279:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 627.076534] LustreError: 15279:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 627.279287] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 627.329137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 627.331534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 631.777163] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403493/real 1789403493] req@ffff9ec32d76f100 x1876325039198336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 631.810755] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 633.158292] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 634.796298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 643.644529] Lustre: DEBUG MARKER: == replay-dual test 2: |X| mkdir adir ==================== 12:32:00 (1789403520) [ 646.997109] LustreError: 16228:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 647.884271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 649.968566] Lustre: Failing over lustre-MDT0000 [ 650.320620] Lustre: server umount lustre-MDT0000 complete [ 667.616278] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403529/real 1789403529] req@ffff9ec201de0e00 x1876325039213952/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789403545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 667.618972] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.641330] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 667.641438] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.651791] Lustre: Skipped 1 previous similar message [ 678.331317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.427698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 679.754238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 680.004142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 680.050709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:161) [ 680.051414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 683.494166] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 693.578469] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 693.590026] Lustre: Skipped 1 previous similar message [ 693.740975] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 695.433757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 703.886881] Lustre: DEBUG MARKER: == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 12:33:00 (1789403580) [ 706.926281] LustreError: 17820:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 707.836266] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 709.938592] Lustre: Failing over lustre-MDT0000 [ 710.242804] Lustre: server umount lustre-MDT0000 complete [ 728.802766] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.156069] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.170362] Lustre: Skipped 1 previous similar message [ 729.439836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.525307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 730.465920] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403592/real 1789403592] req@ffff9ec32d76ed80 x1876325039230720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 730.501789] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 730.992913] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 731.288027] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 731.330092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 731.330298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:193) [ 734.045921] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 734.706454] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 734.714896] Lustre: Skipped 1 previous similar message [ 743.144696] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 744.998740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 753.602563] Lustre: DEBUG MARKER: == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 12:33:50 (1789403630) [ 756.827491] LustreError: 19397:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 757.589617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 759.422133] Lustre: Failing over lustre-MDT0000 [ 759.750636] Lustre: server umount lustre-MDT0000 complete [ 776.674252] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.683977] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.692608] Lustre: Skipped 1 previous similar message [ 786.501827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 791.199618] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 801.048585] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 801.157588] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 801.180885] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:225) [ 801.182604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:225) [ 801.637347] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 801.648825] Lustre: Skipped 1 previous similar message [ 806.395339] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 807.846236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 816.287374] Lustre: DEBUG MARKER: == replay-dual test 5: open, unlink |X| close ============ 12:34:53 (1789403693) [ 816.697716] LustreError: 20008:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255553 [ 816.705775] LustreError: 20008:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255553 [ 820.163064] LustreError: 21000:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 821.108329] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 823.049035] Lustre: Failing over lustre-MDT0000 [ 823.381956] Lustre: server umount lustre-MDT0000 complete [ 841.741096] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 842.116099] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.137657] Lustre: Skipped 2 previous similar messages [ 842.351868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.355835] Lustre: Skipped 1 previous similar message [ 842.470732] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 843.175728] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403705/real 1789403705] req@ffff9ec32e0fc700 x1876325039262976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 843.220665] Lustre: 3306:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 843.577575] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 843.950730] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 843.984648] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:257) [ 843.985542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:257) [ 847.345197] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 847.351206] Lustre: Skipped 1 previous similar message [ 847.727554] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 858.128624] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 859.830424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 869.242817] Lustre: DEBUG MARKER: == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 12:35:46 (1789403746) [ 872.587957] LustreError: 22580:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 873.567129] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 875.632885] Lustre: Failing over lustre-MDT0000 [ 875.996917] Lustre: server umount lustre-MDT0000 complete [ 894.432180] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.438956] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.674488] LustreError: 3303:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ec20329fb80 x1876325039280384/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 905.594826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 910.686659] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 918.812254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 918.861775] LustreError: 23193:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255553 [ 918.872396] LustreError: 23193:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 918.926873] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 918.981811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:289) [ 918.994505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:289) [ 919.598517] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 919.608432] Lustre: Skipped 1 previous similar message [ 924.452877] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 925.729638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 933.665663] Lustre: DEBUG MARKER: == replay-dual test 8: replay of resent request ========== 12:36:50 (1789403810) [ 937.117645] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 937.833267] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 937.834742] LustreError: 23195:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec32d703b80 x1876325019373440/t38654705670(0) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:221/0 lens 512/448 e 0 to 0 dl 1789403826 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 954.134417] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnecting [ 954.171167] Lustre: 23879:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ec32d700e00 x1876325019373440/t38654705670(0) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:238/0 lens 512/2880 e 0 to 0 dl 1789403843 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 957.451941] Lustre: Failing over lustre-MDT0000 [ 957.706411] Lustre: server umount lustre-MDT0000 complete [ 975.840360] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789403838/real 1789403838] req@ffff9ec311519500 x1876325039298048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789403854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 975.879028] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 975.889233] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.905442] Lustre: Skipped 1 previous similar message [ 976.079037] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.632688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.647734] Lustre: Skipped 1 previous similar message [ 976.745988] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 981.131127] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 984.858856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 985.033759] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 985.097708] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:321) [ 985.098795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:321) [ 986.725404] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 986.735866] Lustre: Skipped 2 previous similar messages [ 990.599519] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 991.924500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1000.062974] Lustre: DEBUG MARKER: == replay-dual test 9: resending a replayed create ======= 12:37:57 (1789403877) [ 1003.349809] LustreError: 25888:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1003.353988] LustreError: 25888:0:(osd_handler.c:720:osd_ro()) Skipped 1 previous similar message [ 1004.275697] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1006.840454] Lustre: Failing over lustre-MDT0000 [ 1007.170278] Lustre: server umount lustre-MDT0000 complete [ 1034.724229] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1036.128757] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1036.137633] LustreError: 26596:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec203e5dc00 x1876325019387520/t42949672966(42949672966) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:320/0 lens 520/448 e 0 to 0 dl 1789403925 ref 1 fl Complete:/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 1038.546307] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1052.465458] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [ 1052.611462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:353) [ 1052.621037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:353) [ 1059.301806] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.854455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1069.395915] Lustre: DEBUG MARKER: == replay-dual test 10: resending a replayed unlink ====== 12:39:06 (1789403946) [ 1073.506237] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1076.360320] Lustre: Failing over lustre-MDT0000 [ 1076.614803] Lustre: server umount lustre-MDT0000 complete [ 1098.414478] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1102.695847] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1102.699790] LustreError: 28298:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec311518380 x1876325019403136/t47244640264(47244640264) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:386/0 lens 504/456 e 0 to 0 dl 1789403991 ref 1 fl Complete:/204/0 rc 0/0 job:'unlink.0' uid:0 gid:0 projid:4294967295 [ 1119.039622] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [ 1119.117866] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 2 recovered and 0 were evicted. [ 1119.127273] Lustre: Skipped 1 previous similar message [ 1119.168669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:385) [ 1119.168680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:385) [ 1125.081621] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.775892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1136.639778] Lustre: DEBUG MARKER: == replay-dual test 11: both clients timeout during replay ========================================================== 12:40:13 (1789404013) [ 1140.182864] LustreError: 29322:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1140.190896] LustreError: 29322:0:(osd_handler.c:720:osd_ro()) Skipped 1 previous similar message [ 1141.037880] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1144.148779] Lustre: Failing over lustre-MDT0000 [ 1144.490801] Lustre: server umount lustre-MDT0000 complete [ 1161.120353] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.126746] LustreError: Skipped 2 previous similar messages [ 1161.162499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.173577] Lustre: Skipped 5 previous similar messages [ 1171.431918] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x22bda8e496f6c1ef [ 1171.465808] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 1171.483442] Lustre: Skipped 4 previous similar messages [ 1172.525787] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1172.540604] Lustre: Skipped 1 previous similar message [ 1175.322035] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1175.335305] Lustre: Skipped 2 previous similar messages [ 1175.434767] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1175.448130] LustreError: 30023:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec32d702300 x1876325019416192/t51539607558(51539607558) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:459/0 lens 520/448 e 0 to 0 dl 1789404064 ref 1 fl Complete:/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 1180.320665] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1190.688303] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1190.772686] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 1191.085286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:417) [ 1191.095738] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:417) [ 1193.846506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 1202.784807] Lustre: DEBUG MARKER: == replay-dual test 12: open resend timeout ============== 12:41:19 (1789404079) [ 1207.545244] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1207.737346] LustreError: 29987:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255553 [ 1207.743587] LustreError: 29987:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 1207.749600] LustreError: 29987:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255553 [ 1210.208489] Lustre: Failing over lustre-MDT0000 [ 1210.471968] Lustre: server umount lustre-MDT0000 complete [ 1232.865029] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404095/real 1789404095] req@ffff9ec32d76ed80 x1876325039366912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789404111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1232.885976] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 1238.596243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.604077] Lustre: Skipped 3 previous similar messages [ 1244.409459] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1252.359436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:449) [ 1252.360265] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:449) [ 1258.891296] Lustre: DEBUG MARKER: == replay-dual test 13: close resend timeout ============= 12:42:15 (1789404135) [ 1263.236500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1265.987721] Lustre: Failing over lustre-MDT0000 [ 1266.473678] Lustre: server umount lustre-MDT0000 complete [ 1290.322239] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1293.121385] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 1309.510272] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:23 [ 1309.726793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:481) [ 1309.734187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:481) [ 1316.739893] Lustre: DEBUG MARKER: SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b [ 1318.303387] Lustre: DEBUG MARKER: == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 12:43:15 (1789404195) [ 1322.651361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1325.860204] Lustre: Failing over lustre-MDT0000 [ 1326.244700] Lustre: server umount lustre-MDT0000 complete [ 1352.173409] LustreError: 3303:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ec32d703100 x1876325039399936/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1353.002324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1353.022324] Lustre: Skipped 2 previous similar messages [ 1357.953159] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1425.500195] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1425.504174] Lustre: 34629:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client dd740ea5-06dd-405f-81e9-e270f5cac74b@ [ 1425.522511] Lustre: 34629:0:(genops.c:1600:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1425.536744] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1425.591470] LustreError: 34629:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 1425.598695] LustreError: 34629:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 27 previous similar messages [ 1426.090334] Lustre: lustre-MDT0000: Recovery over after 1:11, of 2 clients 1 recovered and 1 was evicted. [ 1426.101704] Lustre: Skipped 3 previous similar messages [ 1426.151335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:495 to 0x240000400:513) [ 1426.153537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:494 to 0x280000400:513) [ 1432.483799] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.153920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1443.716480] Lustre: DEBUG MARKER: == replay-dual test 15c: remove multiple OST orphans ===== 12:45:20 (1789404320) [ 1446.631485] LustreError: 35620:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1446.637313] LustreError: 35620:0:(osd_handler.c:720:osd_ro()) Skipped 3 previous similar messages [ 1447.323860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1463.798794] LustreError: 34592:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1463.807959] LustreError: 34592:0:(mdt_open.c:646:mdt_finish_open()) Skipped 329 previous similar messages [ 1561.151406] Lustre: Failing over lustre-MDT0000 [ 1561.632560] Lustre: server umount lustre-MDT0000 complete [ 1576.931448] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.950697] Lustre: Skipped 7 previous similar messages [ 1579.416203] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.424598] LustreError: Skipped 3 previous similar messages [ 1580.877986] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1580.887541] Lustre: Skipped 3 previous similar messages [ 1584.193570] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1586.986908] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1586.995806] Lustre: Skipped 8 previous similar messages [ 1650.500213] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1650.511669] Lustre: 36375:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2601fad1-2102-4adf-92e4-4cb577fbb399@ [ 1650.530948] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1650.641118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:494 to 0x280000400:1537) [ 1650.654807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:495 to 0x240000400:1537) [ 1657.247600] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1658.760502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1667.387874] Lustre: DEBUG MARKER: == replay-dual test 16: fail MDS during recovery (3571) == 12:49:04 (1789404544) [ 1671.927212] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1675.222811] Lustre: Failing over lustre-MDT0000 [ 1675.748807] Lustre: server umount lustre-MDT0000 complete [ 1695.723661] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 1695.736101] Lustre: Skipped 1 previous similar message [ 1696.676485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1696.686555] Lustre: Skipped 1 previous similar message [ 1701.493599] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1726.352616] Lustre: Failing over lustre-MDT0000 [ 1726.370788] LustreError: 38492:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1726.380737] Lustre: 38014:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1726.392752] Lustre: 38014:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@ffff9ec207945500 x1876325021829760/t0(73014444037) o101->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:255/0 lens 592/0 e 2 to 0 dl 1789404615 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 1726.406705] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1726.440874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.451446] Lustre: Skipped 2 previous similar messages [ 1726.823687] Lustre: server umount lustre-MDT0000 complete [ 1746.916792] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404609/real 1789404609] req@ffff9ec207946300 x1876325039499008/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789404625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1746.974443] Lustre: 3304:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 36 previous similar messages [ 1757.721797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.726513] Lustre: Skipped 4 previous similar messages [ 1762.939415] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1836.509590] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1836.517793] Lustre: 39000:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c37aecd2-af43-4e29-b6b9-dd9f82a0d797@ [ 1836.536344] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1836.573569] LustreError: 39000:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 1836.582649] LustreError: 39000:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2050 previous similar messages [ 1837.738308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1551 to 0x280000400:1569) [ 1837.738944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1550 to 0x240000400:1569) [ 1844.760740] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1846.779287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1858.656365] Lustre: DEBUG MARKER: == replay-dual test 17: fail OST during recovery (3571) == 12:52:15 (1789404735) [ 1865.216799] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1867.499148] Lustre: Failing over lustre-OST0000 [ 1867.726083] Lustre: server umount lustre-OST0000 complete [ 1868.269875] LustreError: 12914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.308496] LustreError: 12914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1871.137927] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.383713] LustreError: 12914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.250852] LustreError: 35414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.372280] LustreError: 6684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.395583] LustreError: 6684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1886.486574] Lustre: lustre-OST0000: Not available for connect from 192.168.203.7@tcp (not set up) [ 1893.555833] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1918.335049] Lustre: Failing over lustre-OST0000 [ 1918.365517] LustreError: 41157:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 1918.375707] Lustre: 40591:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1918.385245] Lustre: 40591:0:(ldlm_lib.c:2438:target_recovery_overseer()) Skipped 2 previous similar messages [ 1918.392293] LustreError: 40591:0:(ofd_obd.c:1325:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [ 1918.485810] Lustre: server umount lustre-OST0000 complete [ 1927.958108] LustreError: 39700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.979337] LustreError: 39700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1943.304884] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2007.500353] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 2007.503439] Lustre: 41612:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 25761fac-e5f0-4a59-8771-bf039801da39@ [ 2007.533344] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 2007.600781] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 2007.614951] Lustre: Skipped 4 previous similar messages [ 2014.476713] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2015.894274] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2025.702821] Lustre: DEBUG MARKER: == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 12:55:02 (1789404902) [ 2027.757954] LustreError: 38965:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 2027.764796] LustreError: 38965:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1800 previous similar messages [ 2030.016619] LustreError: 38964:0:(ldlm_lockd.c:1361:ldlm_handle_enqueue()) cfs_fail_timeout id 30b sleeping for 40000ms [ 2070.104302] LustreError: 38964:0:(ldlm_lockd.c:1361:ldlm_handle_enqueue()) cfs_fail_timeout id 30b awake [ 2084.040164] Lustre: DEBUG MARKER: == replay-dual test 19: resend of open request =========== 12:56:00 (1789404960) [ 2087.125372] LustreError: 43156:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2087.133246] LustreError: 43156:0:(osd_handler.c:720:osd_ro()) Skipped 2 previous similar messages [ 2087.920426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2089.369928] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2089.372346] LustreError: 39367:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec3180de680 x1876325021933696/t0(0) o101->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:688/0 lens 576/688 e 0 to 0 dl 1789405048 ref 1 fl Interpret:/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 [ 2174.776225] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnecting [ 2178.974493] Lustre: Failing over lustre-MDT0000 [ 2179.377534] Lustre: server umount lustre-MDT0000 complete [ 2196.811834] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.818742] LustreError: Skipped 2 previous similar messages [ 2197.105276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.127589] Lustre: Skipped 6 previous similar messages [ 2201.112539] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2201.369615] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2201.383463] Lustre: Skipped 4 previous similar messages [ 2201.433876] Lustre: 43923:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2201.626739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1584 to 0x240000400:1601) [ 2201.627205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1584 to 0x280000400:1601) [ 2202.606763] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2202.611629] Lustre: Skipped 6 previous similar messages [ 2209.331373] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.781621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2218.316734] Lustre: DEBUG MARKER: == replay-dual test 20: recovery time is not increasing == 12:58:15 (1789405095) [ 2222.084824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2224.075987] Lustre: Failing over lustre-MDT0000 [ 2224.378424] Lustre: server umount lustre-MDT0000 complete [ 2241.972629] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2241.985028] Lustre: Skipped 4 previous similar messages [ 2246.099278] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2383.505374] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2383.519863] Lustre: 45508:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 5055c461-0a4b-45cb-a4d4-a0072f8b6264@ [ 2383.538354] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2383.641598] LustreError: 45508:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 2383.648425] LustreError: 45508:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 55 previous similar messages [ 2383.656678] Lustre: 45508:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2383.665455] Lustre: 45508:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 2 previous similar messages [ 2383.768774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1603 to 0x240000400:1633) [ 2383.772211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1584 to 0x280000400:1633) [ 2390.074444] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.679861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2397.655306] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2399.685295] Lustre: Failing over lustre-MDT0000 [ 2400.158654] Lustre: server umount lustre-MDT0000 complete [ 2416.096811] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405278/real 1789405278] req@ffff9ec207b85f80 x1876325039646720/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789405294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2416.141539] Lustre: 3307:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 2426.883216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.889017] Lustre: Skipped 4 previous similar messages [ 2430.421399] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2568.500171] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2568.514150] Lustre: 46952:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9e1d8fd9-ea3a-4a83-a5f8-cd66488d3d00@ [ 2568.524986] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2568.565700] Lustre: 46952:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2568.574171] Lustre: 46952:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 2 previous similar messages [ 2568.691405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1635 to 0x240000400:1665) [ 2568.698076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1584 to 0x280000400:1665) [ 2573.873811] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2574.997724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2583.553354] Lustre: DEBUG MARKER: == replay-dual test 21a: commit on sharing =============== 13:04:20 (1789405460) [ 2588.573297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2590.473117] Lustre: Failing over lustre-MDT0000 [ 2590.694204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.702336] Lustre: Skipped 1 previous similar message [ 2590.783668] Lustre: server umount lustre-MDT0000 complete [ 2607.911502] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (not set up) [ 2612.779992] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2749.500253] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 2749.504675] Lustre: 48624:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0843fd2d-dc46-4e5a-aacb-2082a314bdd0@ [ 2749.513580] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2749.582096] Lustre: lustre-MDT0000: Recovery over after 2:20, of 2 clients 1 recovered and 1 was evicted. [ 2749.586783] Lustre: Skipped 3 previous similar messages [ 2749.644369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1667 to 0x280000400:1697) [ 2749.644601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1635 to 0x240000400:1697) [ 2757.422595] Lustre: DEBUG MARKER: SKIP: replay-dual test_21b skipping SLOW test 21b [ 2758.770067] Lustre: DEBUG MARKER: == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply [ 2760.036562] Lustre: DEBUG MARKER: SKIP: replay-dual test_22a needs >= 2 MDTs [ 2761.499951] Lustre: DEBUG MARKER: == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply [ 2762.688727] Lustre: DEBUG MARKER: SKIP: replay-dual test_22b needs >= 2 MDTs [ 2764.204813] Lustre: DEBUG MARKER: == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update [ 2765.547533] Lustre: DEBUG MARKER: SKIP: replay-dual test_22c needs >= 2 MDTs [ 2767.002892] Lustre: DEBUG MARKER: == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update [ 2768.053812] Lustre: DEBUG MARKER: SKIP: replay-dual test_22d needs >= 2 MDTs [ 2769.305390] Lustre: DEBUG MARKER: == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 13:07:26 (1789405646) [ 2770.539427] Lustre: DEBUG MARKER: SKIP: replay-dual test_23a needs >= 2 MDTs [ 2772.327186] Lustre: DEBUG MARKER: == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 13:07:29 (1789405649) [ 2773.892585] Lustre: DEBUG MARKER: SKIP: replay-dual test_23b needs >= 2 MDTs [ 2775.822824] Lustre: DEBUG MARKER: == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 13:07:32 (1789405652) [ 2777.169496] Lustre: DEBUG MARKER: SKIP: replay-dual test_23c needs >= 2 MDTs [ 2778.684349] Lustre: DEBUG MARKER: == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 13:07:35 (1789405655) [ 2780.305471] Lustre: DEBUG MARKER: SKIP: replay-dual test_23d needs >= 2 MDTs [ 2781.971950] Lustre: DEBUG MARKER: == replay-dual test 24: reconstruct on non-existing object ========================================================== 13:07:39 (1789405659) [ 2782.060170] LustreError: 48586:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 2782.069298] LustreError: 48586:0:(mdt_open.c:646:mdt_finish_open()) Skipped 9 previous similar messages [ 2783.034897] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2783.041563] LustreError: 49001:0:(ldlm_lib.c:3382:target_send_reply_msg()) @@@ dropping reply req@ffff9ec328c1ed80 x1876325022025728/t94489280518(0) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:626/0 lens 488/456 e 0 to 0 dl 1789405741 ref 1 fl Interpret:/200/0 rc 0/0 job:'truncate.0' uid:0 gid:0 projid:4294967295 [ 2868.014114] Lustre: lustre-MDT0000: Client 956abdbd-640c-41cb-b489-f2b107ffc838 (at 192.168.203.7@tcp) reconnecting [ 2868.039284] Lustre: 48586:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ec32e42bb80 x1876325022025728/t94489280518(0) o36->956abdbd-640c-41cb-b489-f2b107ffc838@192.168.203.7@tcp:711/0 lens 488/3152 e 0 to 0 dl 1789405826 ref 1 fl Interpret:/202/0 rc 0/0 job:'truncate.0' uid:0 gid:0 projid:4294967295 [ 2873.823189] Lustre: DEBUG MARKER: == replay-dual test 25: replay|resend ==================== 13:09:11 (1789405751) [ 2875.522459] Lustre: *** cfs_fail_loc=304, val=0*** [ 2877.416031] Lustre: Failing over lustre-OST0000 [ 2877.575169] Lustre: server umount lustre-OST0000 complete [ 2877.923837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.931577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.945629] Lustre: Skipped 7 previous similar messages [ 2877.951108] LustreError: 39700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.967424] LustreError: 39700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2883.352869] LustreError: 12914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.378734] LustreError: 12914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2888.471577] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.506734] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2894.474944] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2894.478175] Lustre: Skipped 2 previous similar messages [ 2896.050134] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2896.061681] Lustre: Skipped 3 previous similar messages [ 2896.381262] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2896.395517] Lustre: Skipped 7 previous similar messages [ 2901.165447] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2910.775539] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2912.220459] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2920.259603] Lustre: DEBUG MARKER: == replay-dual test 26: dbench and tar with mds failover ========================================================== 13:09:57 (1789405797) [ 2925.916917] LustreError: 52264:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2925.924126] LustreError: 52264:0:(osd_handler.c:720:osd_ro()) Skipped 3 previous similar messages [ 2927.223713] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2931.121103] Lustre: DEBUG MARKER: test_26 fail mds1 1 times [ 2933.716192] Lustre: Failing over lustre-MDT0000 [ 2934.154778] Lustre: server umount lustre-MDT0000 complete [ 2951.520232] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.542805] LustreError: Skipped 3 previous similar messages [ 2960.871122] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x22bda8e496f93cd4 [ 2965.423709] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2969.588338] Lustre: 52964:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2969.594618] Lustre: 52964:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 2 previous similar messages [ 2972.524547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1743 to 0x240000400:1761) [ 2972.527729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1743 to 0x280000400:1761) [ 2979.578700] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2981.151303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2983.880751] LustreError: 53328:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2983.885799] LustreError: 53328:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 259 previous similar messages [ 2989.602286] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2993.087582] Lustre: DEBUG MARKER: test_26 fail mds1 2 times [ 2994.900613] Lustre: Failing over lustre-MDT0000 [ 2995.258419] Lustre: server umount lustre-MDT0000 complete [ 3015.542817] Lustre: 54424:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3015.555253] Lustre: 54424:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 248 previous similar messages [ 3016.984951] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3019.075217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1854 to 0x240000400:1921) [ 3019.077057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1855 to 0x280000400:1889) [ 3020.193163] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789405813/real 1789405813] req@ffff9ec3180ded80 x1876325039771392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789405898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3020.234588] Lustre: 3305:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 3025.801935] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3027.382363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3035.654520] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3039.593650] Lustre: DEBUG MARKER: test_26 fail mds1 3 times [ 3041.852238] Lustre: Failing over lustre-MDT0000 [ 3042.265026] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 3042.386818] Lustre: server umount lustre-MDT0000 complete [ 3069.422895] LustreError: 3303:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ec207a45880 x1876325039914368/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3070.403534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.409285] Lustre: Skipped 4 previous similar messages [ 3074.618611] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3081.032350] Lustre: 55874:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3081.049583] Lustre: 55874:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 347 previous similar messages [ 3084.600957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1984 to 0x240000400:2017) [ 3084.614704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1952 to 0x280000400:1985) [ 3091.955354] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.925942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3102.343250] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3106.505878] Lustre: DEBUG MARKER: test_26 fail mds1 4 times [ 3108.772667] Lustre: Failing over lustre-MDT0000 [ 3109.003523] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 3109.292603] Lustre: server umount lustre-MDT0000 complete [ 3136.999578] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x22bda8e496fab10f [ 3141.514995] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3147.181716] Lustre: 57343:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3147.193747] Lustre: 57343:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 316 previous similar messages [ 3150.134605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2065 to 0x280000400:2081) [ 3150.134606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2096 to 0x240000400:2113) [ 3155.801945] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3157.327273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3210.088438] Lustre: DEBUG MARKER: == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 13:14:46 (1789406086) [ 3236.388360] Lustre: Failing over lustre-OST0000 [ 3236.549898] Lustre: server umount lustre-OST0000 complete [ 3237.345484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.382459] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.408815] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3254.550773] LustreError: 52947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.579498] LustreError: 52947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 3257.579878] Lustre: *** cfs_fail_loc=32a, val=0*** [ 3261.257823] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3271.017622] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3272.472491] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3282.323661] Lustre: DEBUG MARKER: == replay-dual test 29: replay vs update with the same xid ========================================================== 13:15:59 (1789406159) [ 3284.127791] Lustre: DEBUG MARKER: SKIP: replay-dual test_29 needs >= 2 MDTs [ 3286.124418] Lustre: DEBUG MARKER: == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 13:16:02 (1789406162) [ 3289.593804] Lustre: Failing over lustre-MDT0000 [ 3289.941381] Lustre: server umount lustre-MDT0000 complete [ 3290.939481] LustreError: 5828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_convert from 192.168.203.7@tcp arrived at 1789406169 with bad export cookie 2503342667626885496 [ 3322.013387] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3339.772225] Lustre: 60870:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3339.788541] Lustre: 60870:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 308 previous similar messages [ 3339.940243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2149 to 0x240000400:2177) [ 3339.943373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2116 to 0x280000400:2145) [ 3347.974909] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3349.686161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3359.604544] Lustre: DEBUG MARKER: == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 13:17:16 (1789406236) [ 3363.859286] Lustre: Failing over lustre-OST0000 [ 3364.015078] Lustre: server umount lustre-OST0000 complete [ 3365.858132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3365.875810] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.891029] LustreError: 6686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3383.614353] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3383.619509] Lustre: Skipped 7 previous similar messages [ 3383.717365] LustreError: 61869:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 3383.730097] LustreError: 61869:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1286 previous similar messages [ 3388.071345] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3396.884374] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.180431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE [ 3406.589614] Lustre: DEBUG MARKER: == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 13:18:03 (1789406283) [ 3408.218626] Lustre: DEBUG MARKER: SKIP: replay-dual test_32 needs >= 2 MDTs [ 3409.908715] Lustre: DEBUG MARKER: == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 13:18:06 (1789406286) [ 3411.243208] Lustre: DEBUG MARKER: SKIP: replay-dual test_33 ldiskfs only test [ 3412.826427] Lustre: DEBUG MARKER: == replay-dual test complete, duration 3137 sec ========== 13:18:09 (1789406289) [ 3414.165994] Lustre: DEBUG MARKER: === replay-dual: start cleanup 13:18:11 (1789406291) === [ 3421.743484] Lustre: DEBUG MARKER: === replay-dual: finish cleanup 13:18:18 (1789406298) === [ 3423.972841] Lustre: Failing over lustre-MDT0000 [ 3424.489299] Lustre: server umount lustre-MDT0000 complete [ 3443.604319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2116 to 0x280000400:2177) [ 3443.611441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2217 to 0x240000400:2273) [ 3446.961346] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3456.068549] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3457.399484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3463.155374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.491111] Lustre: server umount lustre-MDT0000 complete [ 3468.652898] LustreError: 5826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406346 with bad export cookie 2503342667626978001 [ 3468.663877] LustreError: 5826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3468.761375] Lustre: server umount lustre-OST0000 complete [ 3472.243621] Lustre: server umount lustre-OST0001 complete [ 3482.598338] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3484.866491] Key type lgssc unregistered [ 3485.063711] LNet: 66045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3485.072359] LNetError: 66045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.118936] LNet: Removed LNI 192.168.203.107@tcp [ 3486.881143] Key type .llcrypt unregistered [ 3486.883722] Key type ._llcrypt unregistered