[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474072129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2656MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007344] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010127] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013086] *** VALIDATE selinux *** [ 0.021804] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] *** VALIDATE tmpfs *** [ 0.030439] *** VALIDATE proc *** [ 0.031237] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039781] debug: unmapping init [mem 0xffffffff8e859000-0xffffffff8e860fff] [ 0.042157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043757] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051277] rcu: Hierarchical SRCU implementation. [ 0.053395] smp: Bringing up secondary CPUs ... [ 0.054610] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058476] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.272830] node 0 deferred pages initialised in 207ms [ 0.276122] devtmpfs: initialized [ 0.277235] x86/mm: Memory block size: 128MB [ 0.279825] gcov: version magic: 0x41383552 [ 0.280612] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.283154] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.285450] pinctrl core: initialized pinctrl subsystem [ 0.287234] [ 0.287840] ************************************************************* [ 0.290018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293019] ** ** [ 0.294014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.296019] ** ** [ 0.299019] ** This means that this kernel is built to expose internal ** [ 0.301016] ** IOMMU data structures, which may compromise security on ** [ 0.303021] ** your system. ** [ 0.305020] ** ** [ 0.308017] ** If you see this message and you are not debugging the ** [ 0.310016] ** kernel, report this immediately to your vendor! ** [ 0.312019] ** ** [ 0.314016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.316019] ************************************************************* [ 0.317713] NET: Registered protocol family 16 [ 0.319534] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.322078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.325083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.328133] cpuidle: using governor menu [ 0.329743] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.332603] PCI: Using configuration type 1 for base access [ 0.334118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.343101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.345039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.348147] cryptd: max_cpu_qlen set to 1000 [ 0.352234] ACPI: Added _OSI(Module Device) [ 0.353018] ACPI: Added _OSI(Processor Device) [ 0.355015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.357014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.360466] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.366658] ACPI: Interpreter enabled [ 0.368067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.369014] ACPI: Using IOAPIC for interrupt routing [ 0.371097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.373351] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.382333] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.384043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.387019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.389073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.394292] acpiphp: Slot [2] registered [ 0.395107] acpiphp: Slot [3] registered [ 0.396080] acpiphp: Slot [4] registered [ 0.398122] acpiphp: Slot [5] registered [ 0.400284] acpiphp: Slot [6] registered [ 0.402226] acpiphp: Slot [7] registered [ 0.403178] acpiphp: Slot [8] registered [ 0.405129] acpiphp: Slot [9] registered [ 0.406119] acpiphp: Slot [10] registered [ 0.407220] acpiphp: Slot [11] registered [ 0.409161] acpiphp: Slot [12] registered [ 0.410143] acpiphp: Slot [13] registered [ 0.412351] acpiphp: Slot [14] registered [ 0.414114] acpiphp: Slot [15] registered [ 0.415091] acpiphp: Slot [16] registered [ 0.417108] acpiphp: Slot [17] registered [ 0.418123] acpiphp: Slot [18] registered [ 0.420115] acpiphp: Slot [19] registered [ 0.421092] acpiphp: Slot [20] registered [ 0.422062] acpiphp: Slot [21] registered [ 0.424103] acpiphp: Slot [22] registered [ 0.425110] acpiphp: Slot [23] registered [ 0.427139] acpiphp: Slot [24] registered [ 0.428114] acpiphp: Slot [25] registered [ 0.430076] acpiphp: Slot [26] registered [ 0.431098] acpiphp: Slot [27] registered [ 0.432107] acpiphp: Slot [28] registered [ 0.433107] acpiphp: Slot [29] registered [ 0.435117] acpiphp: Slot [30] registered [ 0.436088] acpiphp: Slot [31] registered [ 0.437060] PCI host bridge to bus 0000:00 [ 0.438014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.442023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.444021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.449027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.451156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459411] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.473011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.475020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.477016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.478013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.481430] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.483663] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.486040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.489672] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.495016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.507018] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.513799] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.517955] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.543020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.564027] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.576546] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.589024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.602029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.621028] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.629000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.640015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.657035] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.667083] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.673015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.701693] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.708025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.715024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.735017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.746408] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.760025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.777020] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.787288] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.790413] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.792385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.794376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.797230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.801101] iommu: Default domain type: Passthrough [ 0.802416] SCSI subsystem initialized [ 0.803108] ACPI: bus type USB registered [ 0.805100] usbcore: registered new interface driver usbfs [ 0.807079] usbcore: registered new interface driver hub [ 0.808068] usbcore: registered new device driver usb [ 0.809166] pps_core: LinuxPPS API ver. 1 registered [ 0.811011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.814060] PTP clock support registered [ 0.816140] EDAC MC: Ver: 3.0.0 [ 0.817451] PCI: Using ACPI for IRQ routing [ 0.818873] NetLabel: Initializing [ 0.819018] NetLabel: domain hash size = 128 [ 0.820010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.821101] NetLabel: unlabeled traffic allowed by default [ 0.823140] vgaarb: loaded [ 0.824281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.825014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.833006] clocksource: Switched to clocksource kvm-clock [ 0.942981] VFS: Disk quotas dquot_6.6.0 [ 0.944877] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.947481] *** VALIDATE ramfs *** [ 0.948723] *** VALIDATE hugetlbfs *** [ 0.950225] pnp: PnP ACPI init [ 0.952739] pnp: PnP ACPI: found 6 devices [ 0.972447] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.976071] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.978614] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.981000] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983300] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985672] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.988350] NET: Registered protocol family 2 [ 0.990797] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.995977] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999235] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.004968] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.007836] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010818] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.012933] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015493] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018563] NET: Registered protocol family 1 [ 1.021344] RPC: Registered named UNIX socket transport module. [ 1.023922] RPC: Registered udp transport module. [ 1.025752] RPC: Registered tcp transport module. [ 1.027531] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.029925] NET: Registered protocol family 44 [ 1.031612] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.033905] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035957] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.038409] PCI: CLS 0 bytes, default 64 [ 1.040389] Unpacking initramfs... [ 2.482736] debug: unmapping init [mem 0xffff96c57cc54000-0xffff96c57ffbffff] [ 2.488477] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.490989] software IO TLB: mapped [mem 0x00000000a2000000-0x00000000a6000000] (64MB) [ 2.495087] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.985946] Initialise system trusted keyrings [ 2.987427] Key type blacklist registered [ 2.989639] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.998658] zbud: loaded [ 3.001629] *** VALIDATE nfs *** [ 3.002922] *** VALIDATE nfs4 *** [ 3.004706] pstore: using deflate compression [ 3.007903] Platform Keyring initialized [ 3.109138] NET: Registered protocol family 38 [ 3.110599] Key type asymmetric registered [ 3.111981] Asymmetric key parser 'x509' registered [ 3.113926] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.117102] io scheduler mq-deadline registered [ 3.118624] io scheduler kyber registered [ 3.120435] io scheduler bfq registered [ 3.122301] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.125066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.127996] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.131186] ACPI: Power Button [PWRF] [ 3.228373] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.322790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.507940] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.605557] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.805279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.834753] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.867568] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.875232] Non-volatile memory driver v1.3 [ 3.876986] Linux agpgart interface v0.103 [ 3.912228] virtio_blk virtio1: [vda] 68272 512-byte logical blocks (35.0 MB/33.3 MiB) [ 3.916331] vda: detected capacity change from 0 to 34955264 [ 3.934792] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.937899] vdb: detected capacity change from 0 to 1073741824 [ 3.954899] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.958490] vdc: detected capacity change from 0 to 2621440000 [ 3.975753] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.978729] vdd: detected capacity change from 0 to 2621440000 [ 3.992880] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.995130] vde: detected capacity change from 0 to 4294967296 [ 4.008188] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.010487] vdf: detected capacity change from 0 to 4294967296 [ 4.015966] libphy: Fixed MDIO Bus: probed [ 4.022777] usbcore: registered new interface driver usbserial_generic [ 4.025307] usbserial: USB Serial support registered for generic [ 4.028830] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.032704] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.034335] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.036719] mousedev: PS/2 mouse device common for all mice [ 4.039811] rtc_cmos 00:05: RTC can wake from S4 [ 4.042260] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.046314] rtc_cmos 00:05: registered as rtc0 [ 4.048476] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.048586] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.054579] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.058085] intel_pstate: CPU model not supported [ 4.061150] hid: raw HID events driver (C) Jiri Kosina [ 4.063281] usbcore: registered new interface driver usbhid [ 4.065378] usbhid: USB HID core driver [ 4.066917] drop_monitor: Initializing network drop monitor service [ 4.069245] Initializing XFRM netlink socket [ 4.071085] NET: Registered protocol family 10 [ 4.073821] Segment Routing with IPv6 [ 4.075919] NET: Registered protocol family 17 [ 4.078127] mpls_gso: MPLS GSO support [ 4.083120] RAS: Correctable Errors collector initialized. [ 4.085592] AVX version of gcm_enc/dec engaged. [ 4.086967] AES CTR mode by8 optimization enabled [ 4.169711] sched_clock: Marking stable (4169687842, 0)->(5100398926, -930711084) [ 4.173432] registered taskstats version 1 [ 4.175481] Loading compiled-in X.509 certificates [ 4.177646] zswap: loaded using pool lzo/zbud [ 4.202127] Key type big_key registered [ 4.213756] Key type encrypted registered [ 4.215630] ima: No TPM chip found, activating TPM-bypass! [ 4.217573] ima: Allocated hash algorithm: sha1 [ 4.219447] ima: No architecture policies found [ 4.221306] evm: Initialising EVM extended attributes: [ 4.223082] evm: security.selinux [ 4.224364] evm: security.ima [ 4.225515] evm: security.capability [ 4.226886] evm: HMAC attrs: 0x1 [ 4.229415] rtc_cmos 00:05: setting system clock to 2025-10-24 13:28:52 UTC (1761312532) [ 4.235713] debug: unmapping init [mem 0xffffffff8f803000-0xffffffff8f9fffff] [ 4.238407] debug: unmapping init [mem 0xffffffff8e582000-0xffffffff8e858fff] [ 4.247175] Write protecting the kernel read-only data: 28672k [ 4.250454] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 4.253593] debug: unmapping init [mem 0xffffffff8d514000-0xffffffff8d5fffff] [ 4.285782] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.294313] systemd[1]: Detected virtualization kvm. [ 4.296171] systemd[1]: Detected architecture x86-64. [ 4.298159] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.322886] systemd[1]: No hostname configured. [ 4.325043] systemd[1]: Set hostname to . [ 4.327298] random: systemd: uninitialized urandom read (16 bytes read) [ 4.329518] systemd[1]: Initializing machine ID from random generator. [ 4.374672] random: ln: uninitialized urandom read (6 bytes read) [ 4.444191] random: systemd: uninitialized urandom read (16 bytes read) [ 4.446551] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.451418] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.456745] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.056355] device-mapper: uevent: version 1.0.3 [ 5.058894] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.805288] virtio_net virtio0 ens2: renamed from eth0 [ 5.836088] random: fast init done [ 5.872270] scsi host0: ata_piix [ 5.876800] scsi host1: ata_piix [ 5.880970] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.883631] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.491495] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.731911] random: crng init done [ 10.733372] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.234183] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.412500] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.692531] SELinux: Disabled at runtime. [ 12.755795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.763471] systemd[1]: Detected virtualization kvm. [ 12.765086] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.685273] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.689234] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.697488] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.702623] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.705216] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.722160] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.732322] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ 14.088167] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.687825] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.280043] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.293884] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.805642] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.030295] EDAC sbridge: Ver: 1.1.2 [ 18.980891] Key type dns_resolver registered [ 19.465200] NFS: Registering the id_resolver key type [ 19.469247] Key type id_resolver registered [ 19.470645] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 47.386462] spl: loading out-of-tree module taints kernel. [ 53.717468] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.767458] alg: No test for adler32 (adler32-zlib) [ 66.531346] Key type ._llcrypt registered [ 66.533192] Key type .llcrypt registered [ 66.654967] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 81.054557] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 82.149454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.733629] Lustre: Lustre: Build Version: 2.15.7_14_g908d8f5 [ 83.389757] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 83.394728] LNet: Accept secure, port 988 [ 85.087182] Key type lgssc registered [ 86.388316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.812924] vdc: vdc1 vdc9 [ 102.404766] vde: vde1 vde9 [ 110.390915] vdf: vdf1 vdf9 [ 123.962449] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 127.323961] hrtimer: interrupt took 8947839 ns [ 129.694488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 129.878835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 129.937996] Lustre: lustre-MDT0000: new disk, initializing [ 130.237640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.278582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.564570] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 139.421877] Lustre: lustre-OST0000: new disk, initializing [ 139.425465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 139.432777] Lustre: Skipped 1 previous similar message [ 139.504103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.762222] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 149.792758] Lustre: lustre-OST0001: new disk, initializing [ 149.798718] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 149.853156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.786286] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 162.140609] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 167.697899] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 176.274464] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.900550] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 183.440262] Lustre: DEBUG MARKER: Client: 2.15.7.14 [ 185.050501] Lustre: DEBUG MARKER: MDS: 2.15.7.14 [ 187.016572] Lustre: DEBUG MARKER: OSS: 2.15.7.14 [ 188.265512] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Oct 24 09:31:55 EDT 2025 [ 194.064672] Lustre: DEBUG MARKER: excepting tests: 59 [ 198.664618] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing check_config_client /mnt/lustre [ 210.414730] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.785963] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 215.769624] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 217.496483] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 09:32:24 (1761312744) [ 220.255268] LustreError: 10643:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 221.094904] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 222.820656] Lustre: Failing over lustre-MDT0000 [ 223.246289] Lustre: server umount lustre-MDT0000 complete [ 233.311433] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312754/real 1761312754] req@00000000876337d9 x1846870132413888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312761 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 233.333241] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.399704] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312754/real 1761312754] req@0000000051aceee5 x1846870132413824/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312761 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 234.399847] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 234.415632] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 234.415727] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 238.505561] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312759/real 1761312759] req@00000000da6a1523 x1846870132414080/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312766 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 238.523808] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 240.543323] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312761/real 1761312761] req@00000000ea7241a6 x1846870132414208/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312768 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 240.544933] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819a9e5 to 0x862565dd1819ae68 [ 240.567613] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 240.907309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.594804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 247.594520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 247.645216] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 250.020074] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 251.042635] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 251.042736] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 251.945376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 257.452772] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 09:33:05 (1761312785) [ 258.756425] Lustre: Failing over lustre-OST0000 [ 258.817733] Lustre: server umount lustre-OST0000 complete [ 261.601170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.614680] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.984921] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.721622] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.840794] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.849470] LustreError: Skipped 1 previous similar message [ 274.146628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.083409] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 276.146836] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 276.147818] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 276.158733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 276.768752] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 283.124409] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 284.210070] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 291.775937] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 09:33:39 (1761312819) [ 294.621752] LustreError: 13682:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 295.489295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 297.164253] Lustre: Failing over lustre-MDT0000 [ 297.535700] Lustre: server umount lustre-MDT0000 complete [ 307.041345] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312828/real 1761312828] req@000000004960b22e x1846870132426688/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312835 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 307.056695] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.081627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.587454] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819ae68 to 0x862565dd1819b7e5 [ 324.597769] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 324.971062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.744136] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 329.242464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 329.249340] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 334.634172] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 339.754282] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 340.453836] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 344.871804] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 349.994699] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 360.235485] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:28 [ 360.244526] Lustre: Skipped 1 previous similar message [ 380.718445] Lustre: lustre-MDT0000: Denying connection for new client 42db78fd-cf1d-4f53-a3a0-714848301871 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:08 [ 380.725769] Lustre: Skipped 3 previous similar messages [ 389.002198] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 389.005294] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 389.047293] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 389.072856] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:129 [ 396.576162] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 09:35:24 (1761312924) [ 398.103600] LustreError: 15120:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 398.673676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 399.901591] Lustre: Failing over lustre-MDT0000 [ 400.178565] Lustre: server umount lustre-MDT0000 complete [ 408.927200] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312930/real 1761312930] req@0000000087e4b9a1 x1846870132438592/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312937 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 408.940491] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 408.944368] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.950143] Lustre: Skipped 1 previous similar message [ 409.040223] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 426.281585] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819b7e5 to 0x862565dd1819bb7a [ 426.290300] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 426.293176] Lustre: Skipped 1 previous similar message [ 426.566093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.624483] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 431.046268] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 431.050360] Lustre: lustre-MDT0000: Denying connection for new client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 431.057833] Lustre: Skipped 1 previous similar message [ 491.001522] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 491.009916] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 491.087627] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 491.111213] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:161 [ 499.778864] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 09:37:07 (1761313027) [ 501.858975] LustreError: 16553:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 502.560750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 504.011514] Lustre: Failing over lustre-MDT0000 [ 504.340573] Lustre: server umount lustre-MDT0000 complete [ 515.487432] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313036/real 1761313036] req@000000005a67fada x1846870132450112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313043 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 515.488224] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.505063] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 515.517769] Lustre: Skipped 1 previous similar message [ 515.525575] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.698626] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819bb7a to 0x862565dd1819bec2 [ 521.705699] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 521.711319] Lustre: Skipped 2 previous similar messages [ 522.088935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.280585] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 531.754514] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 531.816261] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 531.859701] Lustre: lustre-OST0000: deleting orphan objects from 0x0:86 to 0x0:193 [ 536.155895] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.426967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 544.490215] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 09:37:51 (1761313071) [ 547.010529] LustreError: 18185:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 547.943642] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 549.794487] Lustre: Failing over lustre-MDT0000 [ 550.155855] Lustre: server umount lustre-MDT0000 complete [ 560.095171] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313081/real 1761313081] req@000000004654dca0 x1846870132457344/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313088 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 560.119276] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 560.129072] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.502133] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.513969] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819bec2 to 0x862565dd1819c234 [ 570.018335] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 575.786654] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 575.886322] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 575.910718] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 580.199261] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 581.314451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 587.124770] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 09:38:34 (1761313114) [ 588.983245] LustreError: 19824:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 589.681838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 591.168404] Lustre: Failing over lustre-MDT0000 [ 591.439127] Lustre: server umount lustre-MDT0000 complete [ 599.391168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.400864] Lustre: Skipped 1 previous similar message [ 600.415167] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.697490] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819c234 to 0x862565dd1819c60f [ 617.705688] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 617.710816] Lustre: Skipped 5 previous similar messages [ 618.117146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 618.120105] Lustre: Skipped 1 previous similar message [ 618.163846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.842795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.982688] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 620.989040] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 621.018762] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 628.012232] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 629.296944] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 635.816503] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 09:39:23 (1761313163) [ 637.725990] LustreError: 21473:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 638.306816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 639.743046] Lustre: Failing over lustre-MDT0000 [ 640.149881] Lustre: server umount lustre-MDT0000 complete [ 650.591203] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313171/real 1761313171] req@00000000fd68235f x1846870132471872/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313178 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 650.591327] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.618757] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 655.840622] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819c60f to 0x862565dd1819c9a4 [ 656.295287] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.970214] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 665.905972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 666.108225] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 666.197596] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:193 [ 666.207675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:289 [ 671.104731] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 672.664654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 681.013181] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 09:40:08 (1761313208) [ 683.575962] LustreError: 23108:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 684.217549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 695.263160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.263161] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.263169] Lustre: Skipped 3 previous similar messages [ 712.674151] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819c9a4 to 0x862565dd1819ce43 [ 713.093707] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 715.955878] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 726.482403] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:321 [ 726.482529] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:225 [ 730.842664] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.092751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 737.608446] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 09:41:05 (1761313265) [ 738.226145] Lustre: *** cfs_fail_loc=13b, val=315*** [ 738.228228] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 738.230192] LustreError: 23715:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000d9558947 x1846870119864064/t38654705666(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:208/0 lens 392/456 e 0 to 0 dl 1761313283 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 741.570650] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 742.817801] Lustre: Failing over lustre-MDT0000 [ 742.820729] Lustre: Skipped 1 previous similar message [ 743.014300] Lustre: server umount lustre-MDT0000 complete [ 743.019136] Lustre: Skipped 1 previous similar message [ 757.601096] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 757.603845] Lustre: Skipped 8 previous similar messages [ 757.877361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.881051] Lustre: Skipped 2 previous similar messages [ 757.928089] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 760.171425] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 760.175901] Lustre: Skipped 1 previous similar message [ 760.252035] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 760.256983] Lustre: Skipped 1 previous similar message [ 760.276482] Lustre: 25400:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000013c253fd x1846870119864064/t38654705666(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:230/0 lens 392/456 e 0 to 0 dl 1761313305 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 760.285614] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:353 [ 760.293124] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:257 [ 760.669751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 767.248898] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.491273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 774.371915] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 09:41:42 (1761313302) [ 776.162902] LustreError: 26434:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 776.170965] LustreError: 26434:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message [ 776.817543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 785.311315] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313306/real 1761313306] req@00000000545bd926 x1846870132492672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313313 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 785.327831] Lustre: 3014:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 790.498506] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.505586] LustreError: Skipped 1 previous similar message [ 796.647056] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819d26b to 0x862565dd1819d68c [ 796.652115] Lustre: Skipped 1 previous similar message [ 797.147361] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 800.803486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 805.381334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:385 [ 805.389466] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:289 [ 811.092295] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 812.681674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 819.045777] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 09:42:26 (1761313346) [ 821.902769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 822.671475] Lustre: *** cfs_fail_loc=114, val=0*** [ 834.911449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.918268] Lustre: Skipped 5 previous similar messages [ 852.922597] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 856.107766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 865.658342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:417 [ 865.665718] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:321 [ 870.010610] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 871.363940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 877.882875] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 09:43:25 (1761313405) [ 880.972461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 881.655738] Lustre: *** cfs_fail_loc=128, val=0*** [ 884.107284] Lustre: Failing over lustre-MDT0000 [ 884.116149] Lustre: Skipped 2 previous similar messages [ 884.597046] Lustre: server umount lustre-MDT0000 complete [ 884.598960] Lustre: Skipped 2 previous similar messages [ 902.643058] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 906.147917] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 910.643184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 910.657703] Lustre: Skipped 2 previous similar messages [ 910.729444] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 910.733530] Lustre: Skipped 2 previous similar messages [ 910.788715] Lustre: lustre-OST0000: deleting orphan objects from 0x0:259 to 0x0:449 [ 914.873081] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 915.868755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 922.138657] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 09:44:09 (1761313449) [ 924.313726] LustreError: 31540:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 924.321496] LustreError: 31540:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages [ 925.139336] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 935.327378] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.335786] LustreError: Skipped 2 previous similar messages [ 951.776190] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1819df06 to 0x862565dd1819e3c1 [ 951.781835] Lustre: Skipped 2 previous similar messages [ 952.319962] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 954.981077] Lustre: lustre-OST0000: deleting orphan objects from 0x0:460 to 0x0:481 [ 954.981685] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:353 [ 955.645355] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 963.158247] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 964.472598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 972.032680] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 09:44:59 (1761313499) [ 975.403899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1005.104134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:487 to 0x0:513 [ 1005.104314] Lustre: lustre-OST0001: deleting orphan objects from 0x0:359 to 0x0:385 [ 1007.321543] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1014.597758] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1016.079882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1019.367522] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1019.375912] Lustre: Skipped 15 previous similar messages [ 1024.334901] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 09:45:51 (1761313551) [ 1028.483315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1046.879143] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313568/real 1761313568] req@00000000b9d7bebb x1846870132530496/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313575 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1046.897639] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 28 previous similar messages [ 1053.508136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.511717] Lustre: Skipped 5 previous similar messages [ 1053.572862] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1053.595258] Lustre: Skipped 1 previous similar message [ 1056.850045] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1064.190828] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:641 [ 1064.193289] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:513 [ 1069.411966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.442471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1092.446902] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 09:46:59 (1761313619) [ 1096.234792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1106.847372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.855314] Lustre: Skipped 9 previous similar messages [ 1126.275334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:673 [ 1126.275432] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:545 [ 1128.143968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1135.114335] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.400184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1143.992640] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 09:47:51 (1761313671) [ 1146.562807] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1147.792535] Lustre: Failing over lustre-MDT0000 [ 1147.794830] Lustre: Skipped 4 previous similar messages [ 1148.038685] Lustre: server umount lustre-MDT0000 complete [ 1148.043153] Lustre: Skipped 4 previous similar messages [ 1166.449622] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1174.825737] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1174.834352] Lustre: Skipped 4 previous similar messages [ 1174.917227] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1174.925353] Lustre: Skipped 4 previous similar messages [ 1174.956314] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:577 [ 1174.958667] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:705 [ 1178.485877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1179.630322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1185.201112] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 09:48:32 (1761313712) [ 1187.498098] LustreError: 39770:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1187.502378] LustreError: 39770:0:(osd_handler.c:698:osd_ro()) Skipped 4 previous similar messages [ 1188.060255] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1198.047189] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.052317] LustreError: Skipped 4 previous similar messages [ 1215.460566] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd181a7f26 to 0x862565dd181a83a9 [ 1215.469659] Lustre: Skipped 4 previous similar messages [ 1215.473988] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.480992] LustreError: Skipped 1 previous similar message [ 1215.728986] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1215.732448] Lustre: Skipped 2 previous similar messages [ 1218.267078] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1230.222204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:737 [ 1230.222329] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:609 [ 1234.148623] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1235.219639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1242.410388] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 09:49:30 (1761313770) [ 1245.474660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1267.944648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1275.307760] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:769 [ 1275.308446] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:641 [ 1280.301375] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.803351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1288.101524] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 09:50:15 (1761313815) [ 1290.958083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1313.584700] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1320.362820] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:801 [ 1320.362861] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:673 [ 1324.423635] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1325.572360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1331.552023] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 09:50:59 (1761313859) [ 1334.732937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1357.845906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1365.122654] Lustre: lustre-OST0000: deleting orphan objects from 0x0:624 to 0x0:833 [ 1365.124267] Lustre: lustre-OST0001: deleting orphan objects from 0x0:496 to 0x0:705 [ 1368.836431] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1369.895640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1376.077126] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 09:51:43 (1761313903) [ 1378.728515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1405.224525] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.234287] LustreError: Skipped 1 previous similar message [ 1406.616679] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:865 [ 1406.618014] Lustre: lustre-OST0001: deleting orphan objects from 0x0:707 to 0x0:737 [ 1408.594141] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1415.525781] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.469664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1422.811129] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 09:52:30 (1761313950) [ 1425.621182] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1454.537688] Lustre: lustre-OST0001: deleting orphan objects from 0x0:707 to 0x0:769 [ 1454.537832] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:897 [ 1454.954606] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1462.149681] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1463.574762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1470.592750] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 09:53:18 (1761313998) [ 1473.673854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1504.133710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1504.149245] Lustre: Skipped 5 previous similar messages [ 1507.165459] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1520.063121] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1520.063381] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:929 [ 1524.076357] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1525.198520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1531.018473] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 09:54:18 (1761314058) [ 1533.540194] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1534.943952] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.379320] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 1553.382138] Lustre: Skipped 31 previous similar messages [ 1556.670086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1559.929981] Lustre: lustre-OST0001: deleting orphan objects from 0x0:803 to 0x0:833 [ 1559.930035] Lustre: lustre-OST0000: deleting orphan objects from 0x0:835 to 0x0:961 [ 1564.072436] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.333769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1571.903638] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 09:54:59 (1761314099) [ 1575.231128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1584.479191] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314105/real 1761314105] req@00000000701cc241 x1846870132640896/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314112 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1584.491485] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 57 previous similar messages [ 1602.374648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.378060] Lustre: Skipped 10 previous similar messages [ 1605.117264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:993 [ 1605.123113] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 1605.579394] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1612.651835] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1613.889925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1619.041687] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 09:55:46 (1761314146) [ 1621.241333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1630.047193] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.056643] Lustre: Skipped 21 previous similar messages [ 1650.207072] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:897 [ 1650.207072] Lustre: lustre-OST0000: deleting orphan objects from 0x0:995 to 0x0:1025 [ 1651.054405] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1659.171918] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1660.563727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1668.154454] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 09:56:35 (1761314195) [ 1671.028732] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1672.782476] Lustre: Failing over lustre-MDT0000 [ 1672.785322] Lustre: Skipped 10 previous similar messages [ 1673.132502] Lustre: server umount lustre-MDT0000 complete [ 1673.140944] Lustre: Skipped 10 previous similar messages [ 1700.140113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1700.159242] Lustre: Skipped 10 previous similar messages [ 1700.247027] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1700.250635] Lustre: Skipped 10 previous similar messages [ 1700.271685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 1700.271702] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:929 [ 1701.273148] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1708.882298] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1710.214844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1717.705470] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 09:57:25 (1761314245) [ 1720.082694] LustreError: 57814:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1720.086024] LustreError: 57814:0:(osd_handler.c:698:osd_ro()) Skipped 10 previous similar messages [ 1720.977820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1736.095296] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.106434] LustreError: Skipped 10 previous similar messages [ 1742.245046] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd181ab3d7 to 0x862565dd181ab8d8 [ 1742.256801] Lustre: Skipped 10 previous similar messages [ 1745.764581] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1749.590952] Lustre: lustre-OST0001: deleting orphan objects from 0x0:931 to 0x0:961 [ 1749.601868] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 1754.212479] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1755.536313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1762.009528] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 09:58:09 (1761314289) [ 1764.806087] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1798.453861] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1809.850626] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 1809.850689] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 1815.036334] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.212403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1822.309676] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 09:59:09 (1761314349) [ 1825.344558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1847.164795] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1854.006573] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 1854.007408] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1153 [ 1860.485077] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1861.968877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1869.821349] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 09:59:57 (1761314397) [ 1872.904593] Lustre: 62741:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 at adminstrative request [ 1896.725246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1904.003325] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 1904.005086] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1057 [ 1908.447074] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1909.580404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1913.834391] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1924.086180] Lustre: DEBUG MARKER: before 6144, after 6144 [ 1928.558947] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 10:00:56 (1761314456) [ 1929.243472] Lustre: 64784:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 at adminstrative request [ 1935.295236] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:01:03 (1761314463) [ 1937.565381] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1967.429482] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1979.912877] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1188 to 0x0:1217 [ 1979.913839] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 1985.045880] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1986.319060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1993.099144] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:02:00 (1761314520) [ 1996.340073] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2025.952651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2025.961789] Lustre: Skipped 9 previous similar messages [ 2029.331550] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2040.274157] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2040.275209] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2045.760049] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2047.233872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2054.419441] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:03:01 (1761314581) [ 2057.660423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2089.976568] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2100.646914] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2100.647873] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2105.801441] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.145914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2113.551908] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 10:04:01 (1761314641) [ 2116.726748] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2138.109027] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2144.673720] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2144.673990] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2148.454537] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2149.656831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2156.639637] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:04:44 (1761314684) [ 2159.773986] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2179.567535] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 2179.575767] Lustre: Skipped 35 previous similar messages [ 2183.631172] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2188.707697] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2188.711233] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1217 [ 2192.548371] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2193.816771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2200.589602] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:05:28 (1761314728) [ 2203.442700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2211.743215] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314733/real 1761314733] req@00000000ff89d8d7 x1846870132729152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314740 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2211.753750] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 68 previous similar messages [ 2224.424732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.428205] Lustre: Skipped 11 previous similar messages [ 2226.730565] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2233.645851] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2233.646294] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2237.208234] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.213642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2244.862648] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:06:12 (1761314772) [ 2247.859539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2257.247296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.254387] Lustre: Skipped 23 previous similar messages [ 2277.882092] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2277.884229] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2277.969354] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2285.807344] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2287.061882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2293.897175] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:07:01 (1761314821) [ 2296.992297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2298.657590] Lustre: Failing over lustre-MDT0000 [ 2298.660641] Lustre: Skipped 11 previous similar messages [ 2298.964503] Lustre: server umount lustre-MDT0000 complete [ 2298.966502] Lustre: Skipped 11 previous similar messages [ 2327.852063] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2327.860712] Lustre: Skipped 11 previous similar messages [ 2327.985471] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2327.991209] Lustre: Skipped 11 previous similar messages [ 2328.009389] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2328.009394] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2328.798242] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2335.374930] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2336.403185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2342.790568] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:07:50 (1761314870) [ 2345.450340] LustreError: 78350:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2345.453725] LustreError: 78350:0:(osd_handler.c:698:osd_ro()) Skipped 10 previous similar messages [ 2346.228326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2358.754242] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.765745] LustreError: Skipped 11 previous similar messages [ 2364.837945] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd181aefea to 0x862565dd181af4d6 [ 2364.851203] Lustre: Skipped 11 previous similar messages [ 2367.594892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2373.046582] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2373.048282] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 2377.781607] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2378.876703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2384.510220] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:08:32 (1761314912) [ 2386.924079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2406.807300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 2406.809131] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2408.776907] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2415.511112] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2416.571524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2421.841419] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 10:09:09 (1761314949) [ 2424.123682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2450.890273] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1537 [ 2450.893692] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2452.726535] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2458.020552] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2458.880920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2463.743657] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 10:09:51 (1761314991) [ 2464.222138] Lustre: 83172:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 at adminstrative request [ 2471.240657] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 10:09:59 (1761314999) [ 2473.117302] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2479.694509] LustreError: 84049:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2479.698341] LustreError: 84049:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2479.706063] Lustre: 84096:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2479.712924] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2479.810386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1544 to 0x0:1569 [ 2479.813685] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1415 to 0x0:1441 [ 2482.725068] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2492.801856] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 10:10:20 (1761315020) [ 2495.498671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2496.817130] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 2502.814550] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2502.835289] LustreError: 85406:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2502.839730] LustreError: 85406:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2502.845310] Lustre: 85454:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2502.851130] Lustre: 85454:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 2502.854458] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2502.902319] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1580 to 0x0:1601 [ 2502.906330] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1452 to 0x0:1473 [ 2506.108030] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2515.078223] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2520.526462] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 10:10:48 (1761315048) [ 2523.365113] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2530.388349] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.399799] LustreError: Skipped 2 previous similar messages [ 2530.984616] LustreError: 86764:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2530.989355] LustreError: 86764:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2530.997248] Lustre: 86812:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2531.007219] Lustre: 86812:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 2531.011828] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2531.089398] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1480 to 0x0:1505 [ 2531.089430] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1607 to 0x0:1633 [ 2534.378695] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2544.811851] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 10:11:12 (1761315072) [ 2545.641557] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2545.643883] LustreError: 86774:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000081799a58 x1846870120338752/t201863462916(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:494/0 lens 504/456 e 0 to 0 dl 1761315079 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 2555.772346] LustreError: 87983:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2555.777741] LustreError: 87983:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2555.783938] Lustre: 88031:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2555.788383] Lustre: 88031:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 2555.792281] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2555.890289] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1665 [ 2555.890470] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1480 to 0x0:1537 [ 2559.066232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2567.938384] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 10:11:35 (1761315095) [ 2571.219899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2595.421432] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2598.324714] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1667 to 0x0:1697 [ 2598.328357] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1480 to 0x0:1569 [ 2602.577837] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 10:12:10 (1761315130) [ 2605.526693] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2613.641647] LustreError: 90782:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2613.645312] LustreError: 90782:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2613.650414] Lustre: 90828:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2613.654120] Lustre: 90828:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 2613.657295] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2613.711870] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1480 to 0x0:1601 [ 2613.711976] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1667 to 0x0:1729 [ 2616.193102] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2624.586154] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 10:12:32 (1761315152) [ 2640.619120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2660.651418] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2660.654107] Lustre: Skipped 25 previous similar messages [ 2663.207267] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2668.923592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2130 to 0x0:2145 [ 2668.924053] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2002 to 0x0:2017 [ 2673.532559] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2674.676888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2690.767905] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 10:13:38 (1761315218) [ 2703.287579] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2737.285056] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2752.723029] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2433 [ 2752.724079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2546 to 0x0:2561 [ 2757.155859] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2758.284577] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2774.185538] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 10:15:01 (1761315301) [ 2775.436873] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2777.021174] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 10:15:04 (1761315304) [ 2778.672895] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2779.382841] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2779.388930] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2779.403256] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2465 [ 2784.426504] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 10:15:11 (1761315311) [ 2798.960334] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2806.067193] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 2811.183772] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.300045] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.310547] LustreError: Skipped 1 previous similar message [ 2821.416949] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.435602] LustreError: Skipped 1 previous similar message [ 2825.930905] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 2825.941666] Lustre: Skipped 45 previous similar messages [ 2825.942837] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3009 [ 2827.034396] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2881.475921] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 10:16:48 (1761315408) [ 2884.130705] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2896.863364] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315418/real 1761315418] req@00000000e92809c0 x1846870133053568/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315425 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2896.877725] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 64 previous similar messages [ 2914.647107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.650377] Lustre: Skipped 14 previous similar messages [ 2916.364273] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2916.364411] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3041 [ 2917.093465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2919.904260] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.910660] Lustre: Skipped 28 previous similar messages [ 2923.032459] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2924.171565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2940.500611] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 10:17:48 (1761315468) [ 2944.661295] LustreError: 99476:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2950.111164] LustreError: 99476:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2950.114068] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 2950.145633] LustreError: 6314:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 2951.726745] LustreError: 98882:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2956.767155] LustreError: 98882:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2956.777076] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 2958.317601] LustreError: 98883:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2962.401797] LustreError: 98904:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2962.406387] LustreError: 19241:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 2962.417381] LustreError: 19241:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 2962.420722] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2962.420794] LustreError: 98883:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=902 [ 2963.425756] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2881 [ 2964.182432] LustreError: 98883:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2969.388622] LustreError: 100246:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 2969.393297] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 2969.393310] LustreError: 98883:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1 [ 2969.412833] Lustre: Skipped 1 previous similar message [ 2969.415068] Lustre: lustre-MDT0000: Export 000000006a9f310f already connecting from 192.168.203.4@tcp [ 2970.522383] LustreError: 100246:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2975.711129] LustreError: 100246:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2982.367186] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 2982.371204] Lustre: Skipped 1 previous similar message [ 2983.402319] LustreError: 98883:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2983.407113] LustreError: 98883:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 2988.513618] LustreError: 98883:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2988.523408] LustreError: 98883:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3001.827439] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 3001.839908] Lustre: Skipped 2 previous similar messages [ 3003.438574] LustreError: 99476:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3003.443284] LustreError: 99476:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3008.479300] LustreError: 99476:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3008.494457] LustreError: 99476:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3014.146532] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 10:19:02 (1761315542) [ 3015.327632] LustreError: 98883:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3021.621972] Lustre: lustre-MDT0000: Export 000000006a9f310f already connecting from 192.168.203.4@tcp [ 3022.941410] Lustre: lustre-MDT0000: Export 000000006a9f310f already connecting from 192.168.203.4@tcp [ 3025.340072] Lustre: lustre-MDT0000: Export 000000006a9f310f already connecting from 192.168.203.4@tcp [ 3025.343814] Lustre: Skipped 1 previous similar message [ 3030.210229] Lustre: lustre-MDT0000: Export 000000006a9f310f already connecting from 192.168.203.4@tcp [ 3030.213960] Lustre: Skipped 3 previous similar messages [ 3034.522361] LustreError: 98883:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3036.980586] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 3036.984736] Lustre: Skipped 2 previous similar messages [ 3038.649577] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 10:19:26 (1761315566) [ 3041.044146] LustreError: 102553:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3041.048429] LustreError: 102553:0:(osd_handler.c:698:osd_ro()) Skipped 11 previous similar messages [ 3041.681258] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3044.700769] Lustre: Failing over lustre-MDT0000 [ 3044.713905] Lustre: Skipped 13 previous similar messages [ 3045.108549] Lustre: server umount lustre-MDT0000 complete [ 3045.113494] Lustre: Skipped 13 previous similar messages [ 3051.138447] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.146918] LustreError: Skipped 11 previous similar messages [ 3051.152096] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd181e4278 to 0x862565dd181e4b70 [ 3051.160993] Lustre: Skipped 11 previous similar messages [ 3051.338960] Lustre: *** cfs_fail_loc=712, val=0*** [ 3051.341655] LustreError: 19241:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@0000000057115b8a x1846870133077312/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3051.355204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3051.495979] LustreError: 103168:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 3051.506068] LustreError: 103168:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3051.512357] Lustre: 103218:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3051.518347] Lustre: 103218:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 3051.522210] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3051.572637] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3073 [ 3051.578208] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2913 [ 3054.064409] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3087.145157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3087.148468] Lustre: Skipped 8 previous similar messages [ 3087.245073] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3087.265037] Lustre: Skipped 8 previous similar messages [ 3087.319535] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2963 to 0x0:3105 [ 3087.332308] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2945 [ 3089.564991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3097.989435] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3099.304876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3105.270745] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 10:20:33 (1761315633) [ 3105.387301] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 3110.715381] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 10:20:38 (1761315638) [ 3111.381859] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3111.383511] LustreError: 104191:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000eb29fad9 x1846870121374400/t0(0) o700->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:305/0 lens 264/248 e 0 to 0 dl 1761315645 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 3147.737873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 3147.741894] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977 [ 3151.441247] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3159.361475] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3160.674963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3167.578752] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:21:35 (1761315695) [ 3172.832105] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.841416] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.857055] LustreError: Skipped 1 previous similar message [ 3183.402400] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.415658] LustreError: Skipped 4 previous similar messages [ 3187.014768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 3188.745772] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3196.055319] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3197.293507] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3267.956720] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:23:15 (1761315795) [ 3271.607902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3291.563621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3291.567030] Lustre: Skipped 9 previous similar messages [ 3295.120909] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3300.652467] Lustre: *** cfs_fail_loc=216, val=0*** [ 3300.653890] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 3300.655979] LustreError: 109078:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3301.727852] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2998 to 0x0:3041 [ 3365.974639] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 10:24:53 (1761315893) [ 3367.734594] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3367.737703] Lustre: Skipped 2 previous similar messages [ 3367.755714] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3212 to 0x0:3233 [ 3368.521490] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3212 to 0x0:3265 [ 3378.297856] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 10:25:06 (1761315906) [ 3400.740484] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3405.621141] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3405.624573] LustreError: 110759:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000003e19f312 x1846870121422592/t0(0) o101->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:599/0 lens 328/344 e 0 to 0 dl 1761315939 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 3412.778972] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:58 [ 3412.875251] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3073 [ 3412.876045] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3212 to 0x0:3297 [ 3418.023858] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3419.180562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3426.773549] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 10:25:54 (1761315954) [ 3428.937321] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3432.363913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3450.868206] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 3450.871064] Lustre: Skipped 23 previous similar messages [ 3452.850211] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3105 [ 3452.853348] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329 [ 3455.099100] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3462.768562] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3464.128900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3471.704723] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 10:26:39 (1761315999) [ 3472.753647] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3476.807672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3497.284785] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 3497.285923] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3361 [ 3500.127643] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3508.274798] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3510.055918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3517.076507] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 10:27:24 (1761316044) [ 3517.981687] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3522.797453] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3524.393431] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 3524.396466] Lustre: Skipped 1 previous similar message [ 3533.791132] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316055/real 1761316055] req@000000008c3a6830 x1846870133150016/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761316062 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 3533.824559] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 57 previous similar messages [ 3533.830150] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.836824] Lustre: Skipped 18 previous similar messages [ 3552.504021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.507158] Lustre: Skipped 8 previous similar messages [ 3556.271086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3557.147329] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 3557.149083] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169 [ 3565.943706] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 10:28:13 (1761316093) [ 3568.032160] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3568.033746] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3568.042220] LustreError: 116021:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000e485b8d6 x1846870121446720/t261993005073(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:7/0 lens 392/456 e 0 to 0 dl 1761316102 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 3591.532829] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3592.033433] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3201 [ 3592.039325] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 3592.091157] Lustre: 117355:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000004b92bc8b x1846870121446720/t261993005073(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:31/0 lens 392/456 e 0 to 0 dl 1761316126 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 3599.399593] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3600.580371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3607.295159] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 10:28:55 (1761316135) [ 3608.193868] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3608.196278] LustreError: 117353:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000f1e70cc7 x1846870121453440/t266287972368(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:47/0 lens 504/448 e 0 to 0 dl 1761316142 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 3612.288404] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3632.953124] Lustre: 119086:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b580e119 x1846870121453440/t266287972368(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:72/0 lens 504/448 e 0 to 0 dl 1761316167 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 3632.955277] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3457 [ 3632.955731] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 3634.237983] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3640.629205] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3641.550633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3647.750341] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 10:29:35 (1761316175) [ 3648.913437] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3648.918257] LustreError: 119982:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000081b0dbbf x1846870121460608/t270582939664(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:88/0 lens 504/448 e 0 to 0 dl 1761316183 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 3650.623877] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3653.200079] LustreError: 120220:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3653.204111] LustreError: 120220:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages [ 3653.831709] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3655.132758] Lustre: Failing over lustre-MDT0000 [ 3655.135907] Lustre: Skipped 10 previous similar messages [ 3655.343716] Lustre: server umount lustre-MDT0000 complete [ 3655.345398] Lustre: Skipped 10 previous similar messages [ 3663.816190] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.822077] LustreError: Skipped 9 previous similar messages [ 3681.058423] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd181e9115 to 0x862565dd181e973c [ 3681.065935] Lustre: Skipped 9 previous similar messages [ 3684.074160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3688.690959] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3688.697440] Lustre: Skipped 9 previous similar messages [ 3688.759858] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3688.768990] Lustre: Skipped 9 previous similar messages [ 3688.792414] Lustre: 120829:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000075629391 x1846870121460928/t270582939665(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:128/0 lens 392/456 e 0 to 0 dl 1761316223 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 3688.792443] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 3688.792637] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3489 [ 3696.936601] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:30:24 (1761316224) [ 3697.578211] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3697.581057] Lustre: Skipped 1 previous similar message [ 3697.585111] LustreError: 121276:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000009ae02245 x1846870121467456/t274877906960(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:136/0 lens 504/448 e 0 to 0 dl 1761316231 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 3697.596493] LustreError: 121276:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 3699.191740] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3699.198415] Lustre: Skipped 1 previous similar message [ 3701.868151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3722.551560] Lustre: 122510:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000384a4b8a x1846870121467456/t274877906960(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:161/0 lens 504/448 e 0 to 0 dl 1761316256 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 3722.564196] Lustre: 122510:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 3722.566382] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 3722.569584] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3297 [ 3724.184133] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3733.564017] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:31:01 (1761316261) [ 3734.667491] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3736.555823] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3736.565636] LustreError: 122413:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000075c64bc x1846870121473664/t279172874256(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:175/0 lens 392/456 e 0 to 0 dl 1761316270 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 3740.403403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3741.991924] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnecting [ 3742.000039] Lustre: Skipped 1 previous similar message [ 3764.465625] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3764.608826] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3523 to 0x0:3553 [ 3764.612062] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3329 [ 3764.615907] Lustre: 123897:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ec82a6b2 x1846870121473664/t279172874256(0) o35->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:203/0 lens 392/456 e 0 to 0 dl 1761316298 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 3771.606087] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:31:39 (1761316299) [ 3772.217046] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3772.221518] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3772.223910] Lustre: Skipped 1 previous similar message [ 3778.354986] Lustre: 123895:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000c5675766 x1846870121478848/t283467841550(0) o101->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:217/0 lens 664/3424 e 0 to 0 dl 1761316312 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 3783.759295] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:31:51 (1761316311) [ 3786.936325] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3819.173714] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3361 [ 3819.174745] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3555 to 0x0:3585 [ 3820.579894] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3827.228078] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3828.513048] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3845.844180] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:32:53 (1761316373) [ 3850.038933] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3878.191418] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 3878.195760] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3555 to 0x0:3617 [ 3880.159520] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3887.172540] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3888.294208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3892.132755] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3900.593238] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:33:48 (1761316428) [ 3933.669180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3952.210861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3952.214267] Lustre: Skipped 11 previous similar messages [ 3953.790770] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4673 [ 3953.791283] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4868 to 0x0:4897 [ 3956.141101] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3965.772194] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3967.537507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4025.673685] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 10:35:53 (1761316553) [ 4029.418723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4058.597532] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 4058.603510] Lustre: Skipped 32 previous similar messages [ 4060.687610] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4899 to 0x0:4929 [ 4060.693715] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4705 [ 4061.924427] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4068.948322] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4070.247887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4077.563662] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 4078.537306] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4082.631051] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:36:50 (1761316610) [ 4088.812275] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4133.840983] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4133.843786] LustreError: 131554:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000020884a7e x1846870122955776/t300647710728(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:607/0 lens 66040/440 e 0 to 0 dl 1761316702 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 4133.856234] LustreError: 131554:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 4177.235833] Lustre: 131552:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000cdece3ff x1846870122955776/t300647710728(0) o36->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:650/0 lens 66040/440 e 0 to 0 dl 1761316745 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 4183.401159] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4184.435303] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:38:32 (1761316712) [ 4192.079262] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4204.511182] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316725/real 1761316725] req@000000000b24f11c x1846870133579328/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316732 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4204.528866] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 80 previous similar messages [ 4211.111301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.115334] Lustre: Skipped 9 previous similar messages [ 4213.911074] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5030 to 0x0:5057 [ 4213.912736] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4807 to 0x0:4833 [ 4214.962792] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4216.293982] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.316119] Lustre: Skipped 20 previous similar messages [ 4222.673389] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4224.027878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4231.691449] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:39:19 (1761316759) [ 4246.822352] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4256.739838] Lustre: Failing over lustre-OST0000 [ 4256.741585] Lustre: Skipped 7 previous similar messages [ 4256.803510] Lustre: server umount lustre-OST0000 complete [ 4256.805392] Lustre: Skipped 7 previous similar messages [ 4257.254649] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.262852] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.276846] LustreError: Skipped 1 previous similar message [ 4261.195734] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.281059] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.297144] LustreError: Skipped 1 previous similar message [ 4276.348046] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5473 [ 4276.737219] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4291.885437] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.899391] LustreError: Skipped 3 previous similar messages [ 4292.063808] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.240873] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4307.251379] Lustre: Skipped 8 previous similar messages [ 4308.080433] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4308.088607] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5505 [ 4308.092239] Lustre: Skipped 8 previous similar messages [ 4309.020090] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4316.987888] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.406648] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4355.755719] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:41:23 (1761316883) [ 4369.249259] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4369.255789] LustreError: Skipped 7 previous similar messages [ 4375.527788] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd1822e648 to 0x862565dd1823f438 [ 4375.540489] Lustre: Skipped 7 previous similar messages [ 4377.028542] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5249 [ 4377.033214] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5537 [ 4379.580073] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4421.799452] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5281 [ 4423.322169] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4429.950859] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4431.149549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4437.181766] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:42:44 (1761316964) [ 4451.297946] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.317241] LustreError: Skipped 6 previous similar messages [ 4471.711795] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4476.254747] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5569 [ 4481.768127] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4483.662423] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4493.051099] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:43:40 (1761317020) [ 4502.959915] Lustre: *** cfs_fail_loc=605, val=0*** [ 4502.961623] LustreError: 142558:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000000da48507 failed: rc = -95 [ 4502.973125] LustreError: 142558:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 4502.980670] LustreError: 142558:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 4502.986566] LustreError: 142558:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 4502.991281] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4502.996902] LustreError: 142558:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 4503.091756] LustreError: 142558:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4513.921140] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5458 to 0x0:5601 [ 4513.921607] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5284 to 0x0:5313 [ 4516.966819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4524.144476] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 10:44:11 (1761317051) [ 4526.754206] LustreError: 143610:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4526.758505] LustreError: 143610:0:(osd_handler.c:698:osd_ro()) Skipped 8 previous similar messages [ 4527.632631] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4549.875478] Lustre: *** cfs_fail_loc=707, val=0*** [ 4551.205660] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4590.420320] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:58 [ 4591.017659] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5615 to 0x0:5633 [ 4591.017684] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5326 to 0x0:5345 [ 4595.656858] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4596.941077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4603.852549] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 10:45:31 (1761317131) [ 4632.239424] LustreError: 144252:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 4638.295136] LustreError: 144252:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4653.368080] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:46:21 (1761317181) [ 4680.975262] LustreError: 6322:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 4687.023898] LustreError: 6322:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 4697.070935] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:47:04 (1761317224) [ 4724.547103] LustreError: 144706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4729.647144] LustreError: 144706:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4731.195733] LustreError: 144706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 4741.287234] LustreError: 144706:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4758.134935] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:48:05 (1761317285) [ 4819.474896] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:49:06 (1761317346) [ 4849.332325] LustreError: 144706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 4849.767165] LustreError: 144706:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4857.524093] LustreError: 144254:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 4857.535690] LustreError: 144254:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 20 previous similar messages [ 4857.967182] LustreError: 144254:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4857.973344] LustreError: 144254:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 20 previous similar messages [ 4873.949863] LustreError: 144706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 4873.965373] LustreError: 144706:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 48 previous similar messages [ 4874.383143] LustreError: 144706:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 4874.389130] LustreError: 144706:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 48 previous similar messages [ 4882.093492] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:50:09 (1761317409) [ 4914.253645] Lustre: DEBUG MARKER: phase 2 [ 4922.225483] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 10:50:49 (1761317449) [ 5001.996451] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 10:52:09 (1761317529) [ 5003.345813] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5004.953856] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 10:52:12 (1761317532) [ 5007.710537] Lustre: DEBUG MARKER: Started rundbench load pid=131180 ... [ 5012.134666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5014.446521] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5016.446095] Lustre: Failing over lustre-MDT0000 [ 5016.449310] Lustre: Skipped 6 previous similar messages [ 5016.855489] Lustre: server umount lustre-MDT0000 complete [ 5016.860504] Lustre: Skipped 7 previous similar messages [ 5026.143208] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317547/real 1761317547] req@0000000087d2e4e6 x1846870133764288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317554 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5026.159450] Lustre: 3013:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 55 previous similar messages [ 5026.164793] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.184771] Lustre: Skipped 11 previous similar messages [ 5027.231166] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.244595] LustreError: Skipped 3 previous similar messages [ 5044.514783] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd182403c7 to 0x862565dd182459d4 [ 5044.525338] Lustre: Skipped 3 previous similar messages [ 5044.533050] Lustre: MGC192.168.203.104@tcp: Connection restored to (at 0@lo) [ 5044.537069] Lustre: Skipped 20 previous similar messages [ 5044.899429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.903424] Lustre: Skipped 7 previous similar messages [ 5044.965731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5044.975662] Lustre: Skipped 9 previous similar messages [ 5047.794566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5058.924388] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5058.929062] Lustre: Skipped 5 previous similar messages [ 5059.495579] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5059.504049] Lustre: Skipped 5 previous similar messages [ 5059.533476] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5735 to 0x0:5761 [ 5059.535282] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 5063.682586] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5064.845663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5070.479990] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5072.929486] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 5074.547624] LustreError: 152163:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export 00000000e573c509 ns: mdt-lustre-MDT0000_UUID lock: 0000000072869b57/0x862565dd1824b1a8 lrc: 4/0,0 mode: PR/PR res: [0x20001b1b3:0xf3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.4@tcp remote: 0xb7a9e08542ef8dbe expref: 3 pid: 152163 timeout: 0 lvb_type: 0 [ 5075.424403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.505585] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5469 to 0x0:5505 [ 5108.517257] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5790 to 0x0:5825 [ 5109.221982] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5117.354364] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5118.761068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5125.527908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5128.251742] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5130.247814] LustreError: 153377:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export 00000000884baf87 ns: mdt-lustre-MDT0000_UUID lock: 00000000232af783/0x862565dd1825233e lrc: 3/0,0 mode: PR/PR res: [0x20001b1b3:0xf95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.4@tcp remote: 0xb7a9e08542efad84 expref: 3 pid: 153377 timeout: 0 lvb_type: 0 [ 5160.726751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5160.888876] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5550 to 0x0:5569 [ 5160.894029] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5871 to 0x0:5889 [ 5167.101984] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5168.040931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5173.733116] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 10:55:01 (1761317701) [ 5296.833864] LustreError: 155874:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5296.839117] LustreError: 155874:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages [ 5297.504836] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5309.249940] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 5339.063448] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5345.915330] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6639 to 0x0:6657 [ 5345.915479] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6318 to 0x0:6337 [ 5351.259143] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5352.694855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5410.048812] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 10:58:57 (1761317937) [ 5410.986384] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 5412.462736] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:58:59 (1761317939) [ 5413.790541] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 5414.965198] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 10:59:02 (1761317942) [ 5421.017583] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5423.195903] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5427.514366] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.522270] LustreError: Skipped 3 previous similar messages [ 5428.195715] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.620730] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.630101] LustreError: Skipped 1 previous similar message [ 5441.813360] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6834 to 0x0:6849 [ 5443.252563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5450.659631] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5451.820288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5461.526181] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5463.959255] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5465.569719] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.579474] LustreError: Skipped 3 previous similar messages [ 5482.424666] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6834 to 0x0:6881 [ 5485.307315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5495.224086] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5497.242415] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5505.816886] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 11:00:33 (1761318033) [ 5506.785232] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 5507.967038] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 11:00:35 (1761318035) [ 5510.851402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5530.932231] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5533.731577] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5538.110082] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:57 [ 5538.196182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6513 to 0x0:6529 [ 5538.196836] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6883 to 0x0:6913 [ 5541.889462] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5542.881555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5548.785500] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 11:01:16 (1761318076) [ 5551.508463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5571.244062] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5571.246332] LustreError: 163415:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000002204deb x1846870127650048/t343597383683(343597383683) o101->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:500/0 lens 592/600 e 0 to 0 dl 1761318105 ref 1 fl Interpret:/4/0 rc 301/0 job:'multiop.0' [ 5572.345517] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5577.543805] Lustre: lustre-MDT0000: Client 13d7bc1c-a67c-47a9-b17d-f86bf9f61702 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:59 [ 5577.565404] Lustre: 163414:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000003018036f x1846870127650048/t343597383683(343597383683) o101->13d7bc1c-a67c-47a9-b17d-f86bf9f61702@192.168.203.4@tcp:506/0 lens 592/3424 e 0 to 0 dl 1761318111 ref 1 fl Interpret:/6/0 rc 0/0 job:'multiop.0' [ 5577.715176] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6561 [ 5577.715623] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6883 to 0x0:6945 [ 5581.910685] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5583.314383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5590.610414] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 11:01:58 (1761318118) [ 5593.055878] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -19 [ 5595.632516] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.641158] LustreError: Skipped 5 previous similar messages [ 5625.894031] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6593 [ 5628.866141] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5630.652749] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.662875] LustreError: Skipped 1 previous similar message [ 5635.407988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6883 to 0x0:6977 [ 5636.248587] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5642.993151] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 11:02:50 (1761318170) [ 5643.836281] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 5644.883503] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 11:02:52 (1761318172) [ 5645.692693] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 5646.780528] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 11:02:54 (1761318174) [ 5647.637729] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 5648.588753] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 11:02:56 (1761318176) [ 5649.380144] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 5650.409903] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 11:02:58 (1761318178) [ 5651.328451] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 5652.562172] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 11:03:00 (1761318180) [ 5653.477587] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 5654.680969] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 11:03:02 (1761318182) [ 5655.835525] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 5657.156480] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 11:03:04 (1761318184) [ 5658.332825] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 5659.693826] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 11:03:07 (1761318187) [ 5660.747507] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 5661.960878] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 11:03:09 (1761318189) [ 5663.094836] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 5664.119410] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 11:03:12 (1761318192) [ 5664.829638] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 5665.922402] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 11:03:13 (1761318193) [ 5666.913181] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 5668.140621] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 11:03:15 (1761318195) [ 5669.151275] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 5670.355632] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 11:03:18 (1761318198) [ 5671.497686] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 5672.658111] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 11:03:20 (1761318200) [ 5673.681734] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 5674.914403] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 11:03:22 (1761318202) [ 5676.360350] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 5677.659455] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 11:03:25 (1761318205) [ 5678.941084] Lustre: 168029:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d7919cce-e5be-4d6a-b783-288094b85988 at adminstrative request [ 5687.680791] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 11:03:35 (1761318215) [ 5693.341624] Lustre: Failing over lustre-MDT0000 [ 5693.345743] Lustre: Skipped 9 previous similar messages [ 5693.731209] Lustre: server umount lustre-MDT0000 complete [ 5693.733025] Lustre: Skipped 9 previous similar messages [ 5705.120787] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318226/real 1761318226] req@0000000073a0490f x1846870134297216/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318233 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 5705.137728] Lustre: 3015:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 32 previous similar messages [ 5705.141589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.146783] Lustre: Skipped 15 previous similar messages [ 5705.183391] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5705.191642] LustreError: Skipped 6 previous similar messages [ 5711.331178] Lustre: Evicted from MGS (at 192.168.203.104@tcp) after server handle changed from 0x862565dd182ca1bc to 0x862565dd182cc48b [ 5711.339880] Lustre: Skipped 6 previous similar messages [ 5711.345192] Lustre: MGC192.168.203.104@tcp: Connection restored to 192.168.203.104@tcp (at 0@lo) [ 5711.351032] Lustre: Skipped 22 previous similar messages [ 5711.682990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5711.685886] Lustre: Skipped 9 previous similar messages [ 5711.736268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5711.739557] Lustre: Skipped 8 previous similar messages [ 5714.758708] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5721.902219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 5721.908810] Lustre: Skipped 8 previous similar messages [ 5722.480309] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 5722.485350] Lustre: Skipped 8 previous similar messages [ 5722.525095] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7029 to 0x0:7073 [ 5722.525189] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6645 to 0x0:6689 [ 5726.609349] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5728.138563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5735.870570] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 11:04:23 (1761318263) [ 5752.635226] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5752.643465] LustreError: Skipped 1 previous similar message [ 5767.738795] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7174 to 0x0:7201 [ 5769.338161] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5776.628188] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5777.631165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5784.680612] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 11:05:12 (1761318312) [ 5795.207182] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7174 to 0x0:7233 [ 5795.207182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6645 to 0x0:6721 [ 5798.555719] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5805.092175] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 11:05:32 (1761318332) [ 5808.799222] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5829.002202] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7265 [ 5830.381685] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5838.321217] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5839.547625] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5846.267966] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 11:06:13 (1761318373) [ 5850.134126] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5854.689074] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5871.202786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7297 [ 5872.763115] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5880.185945] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5881.442402] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5888.545237] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 11:06:56 (1761318416) [ 5892.372703] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5895.666844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5905.148786] LustreError: 7821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318433 with bad export cookie 9666244175488416830 [ 5938.366895] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.391199] LustreError: Skipped 19 previous similar messages [ 5942.300045] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5944.874924] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6763 to 0x0:6785 [ 5963.128450] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5967.178256] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7235 to 0x0:7329 [ 5979.386799] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 11:08:26 (1761318506) [ 5995.311665] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 5995.316239] Lustre: Skipped 1 previous similar message [ 6030.902550] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6040.575250] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6763 to 0x0:6817 [ 6047.977023] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6049.980672] Lustre: lustre-OST0000: Denying connection for new client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 6049.988289] Lustre: Skipped 11 previous similar messages [ 6060.339468] Lustre: lustre-OST0000: Denying connection for new client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 6060.358625] Lustre: Skipped 1 previous similar message [ 6080.817381] Lustre: lustre-OST0000: Denying connection for new client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 6080.845800] Lustre: Skipped 3 previous similar messages [ 6115.003498] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6115.007935] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6115.085454] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7340 to 0x0:7361 [ 6118.904417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec [ 6131.552282] Lustre: DEBUG MARKER: free_before: 7518208 free_after: 7518208 [ 6136.790966] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 11:11:04 (1761318664) [ 6140.897645] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6159.658226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7364 to 0x0:7393 [ 6161.604537] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6169.866871] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 11:11:37 (1761318697) [ 6175.201385] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6175.216947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6194.531374] Lustre: *** cfs_fail_loc=715, val=0*** [ 6194.941319] LustreError: 182906:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6194.945530] LustreError: 182906:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 15 previous similar messages [ 6195.999343] Lustre: *** cfs_fail_loc=715, val=0*** [ 6196.317089] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6200.619410] Lustre: lustre-OST0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 6201.695148] Lustre: *** cfs_fail_loc=715, val=0*** [ 6202.335508] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 0:56 [ 6207.791572] Lustre: lustre-OST0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 2 clients in recovery for 0:51 [ 6208.863252] Lustre: *** cfs_fail_loc=715, val=0*** [ 6208.865018] Lustre: Skipped 1 previous similar message [ 6214.952104] Lustre: lustre-OST0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 2 clients in recovery for 0:44 [ 6214.973873] Lustre: Skipped 1 previous similar message [ 6216.032227] Lustre: *** cfs_fail_loc=715, val=0*** [ 6216.036763] Lustre: Skipped 1 previous similar message [ 6222.123375] Lustre: lustre-OST0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [ 6222.139516] Lustre: Skipped 1 previous similar message [ 6224.223476] Lustre: *** cfs_fail_loc=715, val=0*** [ 6224.225451] Lustre: Skipped 2 previous similar messages [ 6235.039301] LustreError: 182906:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6235.042204] LustreError: 182906:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 14 previous similar messages [ 6235.096488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7425 [ 6239.341762] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6240.338439] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6246.921833] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 11:12:54 (1761318774) [ 6269.096868] LustreError: 184457:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6270.111083] Lustre: *** cfs_fail_loc=715, val=0*** [ 6270.113105] Lustre: Skipped 2 previous similar messages [ 6270.448891] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6275.371503] Lustre: lustre-MDT0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:59 [ 6275.379120] Lustre: Skipped 3 previous similar messages [ 6296.875866] Lustre: lustre-MDT0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:38 [ 6296.893819] Lustre: Skipped 2 previous similar messages [ 6305.055375] Lustre: *** cfs_fail_loc=715, val=0*** [ 6305.058637] Lustre: Skipped 4 previous similar messages [ 6332.711545] Lustre: lustre-MDT0000: Client 1e12213b-3e1e-4e9f-8c7b-217b27abad11 (at 192.168.203.4@tcp) reconnected, waiting for 1 clients in recovery for 0:02 [ 6332.723687] Lustre: Skipped 4 previous similar messages [ 6339.879492] Lustre: lustre-MDT0000: Recovery already passed deadline 0:04. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 6347.057205] Lustre: lustre-MDT0000: Recovery already passed deadline 0:12. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 6349.151247] LustreError: 184457:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6349.193608] Lustre: 184457:0:(ldlm_lib.c:2829:target_recovery_thread()) too long recovery - read logs [ 6349.198850] LustreError: dumping log to /tmp/lustre-log.1761318877.184457 [ 6349.272753] Lustre: lustre-MDT0000: Recovery over after 1:20, of 1 clients 1 recovered and 0 were evicted. [ 6349.279552] Lustre: Skipped 9 previous similar messages [ 6349.316386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7436 to 0x0:7457 [ 6349.317326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6830 to 0x0:6849 [ 6353.728247] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6355.016092] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6360.846099] Lustre: DEBUG MARKER: == replay-single test complete, duration 6171 sec ======== 11:14:48 (1761318888) [ 6370.223385] Lustre: server umount lustre-MDT0000 complete [ 6370.226460] Lustre: Skipped 11 previous similar messages [ 6373.147696] LustreError: 7821:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318901 with bad export cookie 9666244175488429157 [ 6373.154047] LustreError: 166-1: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.161887] LustreError: Skipped 4 previous similar messages [ 6377.887143] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318899/real 1761318899] req@000000006a59138a x1846870134413184/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318906 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 6377.913295] Lustre: 3016:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [ 6377.931206] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.948474] Lustre: Skipped 13 previous similar messages [ 6390.348517] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6391.928230] Key type lgssc unregistered [ 6392.100508] LNet: 186464:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6392.104790] LNet: Removed LNI 192.168.203.104@tcp [ 6392.483411] Key type .llcrypt unregistered [ 6392.485502] Key type ._llcrypt unregistered