[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 554288888 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2866568K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007353] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.013017] SELinux: Initializing. [ 0.014096] *** VALIDATE selinux *** [ 0.023090] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027734] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030128] *** VALIDATE tmpfs *** [ 0.032369] *** VALIDATE proc *** [ 0.033253] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036184] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042005] debug: unmapping init [mem 0xffffffff87059000-0xffffffff87060fff] [ 0.044929] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045773] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.048909] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.050908] ... event mask: 000000070000000f [ 0.051292] rcu: Hierarchical SRCU implementation. [ 0.053384] smp: Bringing up secondary CPUs ... [ 0.054643] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.058144] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.119673] node 0 deferred pages initialised in 55ms [ 0.123078] devtmpfs: initialized [ 0.124218] x86/mm: Memory block size: 128MB [ 0.127637] gcov: version magic: 0x41383552 [ 0.130127] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133347] pinctrl core: initialized pinctrl subsystem [ 0.135184] [ 0.135728] ************************************************************* [ 0.137013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139018] ** ** [ 0.141011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143010] ** ** [ 0.145010] ** This means that this kernel is built to expose internal ** [ 0.147009] ** IOMMU data structures, which may compromise security on ** [ 0.149019] ** your system. ** [ 0.151011] ** ** [ 0.153013] ** If you see this message and you are not debugging the ** [ 0.155014] ** kernel, report this immediately to your vendor! ** [ 0.157017] ** ** [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161019] ************************************************************* [ 0.163725] NET: Registered protocol family 16 [ 0.165466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174078] cpuidle: using governor menu [ 0.175845] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178463] PCI: Using configuration type 1 for base access [ 0.181125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193077] cryptd: max_cpu_qlen set to 1000 [ 0.195106] ACPI: Added _OSI(Module Device) [ 0.196007] ACPI: Added _OSI(Processor Device) [ 0.197008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201694] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206558] ACPI: Interpreter enabled [ 0.208069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.209011] ACPI: Using IOAPIC for interrupt routing [ 0.211107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223382] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233530] acpiphp: Slot [2] registered [ 0.235137] acpiphp: Slot [5] registered [ 0.237149] acpiphp: Slot [6] registered [ 0.238123] acpiphp: Slot [7] registered [ 0.240160] acpiphp: Slot [8] registered [ 0.242127] acpiphp: Slot [9] registered [ 0.243105] acpiphp: Slot [10] registered [ 0.245095] acpiphp: Slot [3] registered [ 0.246136] acpiphp: Slot [4] registered [ 0.247096] acpiphp: Slot [11] registered [ 0.248097] acpiphp: Slot [12] registered [ 0.249101] acpiphp: Slot [13] registered [ 0.250069] acpiphp: Slot [14] registered [ 0.250933] acpiphp: Slot [15] registered [ 0.252085] acpiphp: Slot [16] registered [ 0.253078] acpiphp: Slot [17] registered [ 0.254114] acpiphp: Slot [18] registered [ 0.254984] acpiphp: Slot [19] registered [ 0.256053] acpiphp: Slot [20] registered [ 0.256923] acpiphp: Slot [21] registered [ 0.258086] acpiphp: Slot [22] registered [ 0.260097] acpiphp: Slot [23] registered [ 0.261097] acpiphp: Slot [24] registered [ 0.262108] acpiphp: Slot [25] registered [ 0.264111] acpiphp: Slot [26] registered [ 0.266089] acpiphp: Slot [27] registered [ 0.267060] acpiphp: Slot [28] registered [ 0.268063] acpiphp: Slot [29] registered [ 0.268971] acpiphp: Slot [30] registered [ 0.270068] acpiphp: Slot [31] registered [ 0.270962] PCI host bridge to bus 0000:00 [ 0.271015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.280020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283856] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285978] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.297040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304630] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307713] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.311055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.314558] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.318014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.328014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.331010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334777] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.346018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.358017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365773] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.370015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.374015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.385017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.393131] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.401017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.410018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.423018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.435000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.440015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.445014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.461019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.470762] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.477020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.482015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.494134] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.504772] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.511018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.517016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.531015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.538000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.541424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.544500] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.546476] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.549352] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.553286] iommu: Default domain type: Passthrough [ 0.555295] SCSI subsystem initialized [ 0.557137] ACPI: bus type USB registered [ 0.558098] usbcore: registered new interface driver usbfs [ 0.560087] usbcore: registered new interface driver hub [ 0.561081] usbcore: registered new device driver usb [ 0.563160] pps_core: LinuxPPS API ver. 1 registered [ 0.565011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.567058] PTP clock support registered [ 0.569131] EDAC MC: Ver: 3.0.0 [ 0.571124] PCI: Using ACPI for IRQ routing [ 0.573833] NetLabel: Initializing [ 0.575015] NetLabel: domain hash size = 128 [ 0.576010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.578088] NetLabel: unlabeled traffic allowed by default [ 0.580369] vgaarb: loaded [ 0.581293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.584015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.592000] clocksource: Switched to clocksource kvm-clock [ 0.692119] VFS: Disk quotas dquot_6.6.0 [ 0.693587] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.696223] *** VALIDATE ramfs *** [ 0.697547] *** VALIDATE hugetlbfs *** [ 0.699153] pnp: PnP ACPI init [ 0.701559] pnp: PnP ACPI: found 6 devices [ 0.715969] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.718685] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721029] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.723172] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.725575] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.728168] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.731333] NET: Registered protocol family 2 [ 0.733924] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.738882] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.742971] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.749623] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.753259] TCP: Hash tables configured (established 65536 bind 65536) [ 0.756242] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.759703] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762926] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766155] NET: Registered protocol family 1 [ 0.769301] RPC: Registered named UNIX socket transport module. [ 0.771484] RPC: Registered udp transport module. [ 0.773350] RPC: Registered tcp transport module. [ 0.775212] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.777567] NET: Registered protocol family 44 [ 0.779194] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.781448] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.783636] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.786157] PCI: CLS 0 bytes, default 64 [ 0.787415] Unpacking initramfs... [ 2.117268] debug: unmapping init [mem 0xffff9af97cc54000-0xffff9af97ffbffff] [ 2.121297] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.122981] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.125614] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.590322] Initialise system trusted keyrings [ 2.592136] Key type blacklist registered [ 2.593857] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.601880] zbud: loaded [ 2.604248] *** VALIDATE nfs *** [ 2.605475] *** VALIDATE nfs4 *** [ 2.607312] pstore: using deflate compression [ 2.610706] Platform Keyring initialized [ 2.719761] NET: Registered protocol family 38 [ 2.722657] Key type asymmetric registered [ 2.724518] Asymmetric key parser 'x509' registered [ 2.726431] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.729623] io scheduler mq-deadline registered [ 2.731442] io scheduler kyber registered [ 2.733280] io scheduler bfq registered [ 2.735231] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.739359] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.741866] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.744743] ACPI: Power Button [PWRF] [ 2.750230] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.757203] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.788919] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.801975] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.819365] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.849109] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.877809] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.883118] Non-volatile memory driver v1.3 [ 2.884871] Linux agpgart interface v0.103 [ 2.913154] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.915914] vda: detected capacity change from 0 to 74706944 [ 2.944173] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.947794] vdb: detected capacity change from 0 to 1073741824 [ 2.963902] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.967063] vdc: detected capacity change from 0 to 2621440000 [ 2.982163] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.985097] vdd: detected capacity change from 0 to 2621440000 [ 2.999470] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.002734] vde: detected capacity change from 0 to 4294967296 [ 3.017205] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.020453] vdf: detected capacity change from 0 to 4294967296 [ 3.026983] libphy: Fixed MDIO Bus: probed [ 3.032500] usbcore: registered new interface driver usbserial_generic [ 3.034601] usbserial: USB Serial support registered for generic [ 3.036307] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.040234] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.042134] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.044735] mousedev: PS/2 mouse device common for all mice [ 3.047946] rtc_cmos 00:05: RTC can wake from S4 [ 3.051149] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.051310] rtc_cmos 00:05: registered as rtc0 [ 3.056332] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.057910] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.059193] intel_pstate: CPU model not supported [ 3.064213] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.069068] hid: raw HID events driver (C) Jiri Kosina [ 3.071295] usbcore: registered new interface driver usbhid [ 3.073469] usbhid: USB HID core driver [ 3.075275] drop_monitor: Initializing network drop monitor service [ 3.077790] Initializing XFRM netlink socket [ 3.079872] NET: Registered protocol family 10 [ 3.083243] Segment Routing with IPv6 [ 3.084737] NET: Registered protocol family 17 [ 3.086783] mpls_gso: MPLS GSO support [ 3.092436] RAS: Correctable Errors collector initialized. [ 3.094605] AVX version of gcm_enc/dec engaged. [ 3.096335] AES CTR mode by8 optimization enabled [ 3.171451] sched_clock: Marking stable (3171410344, 0)->(4068985221, -897574877) [ 3.175550] registered taskstats version 1 [ 3.177579] Loading compiled-in X.509 certificates [ 3.180541] zswap: loaded using pool lzo/zbud [ 3.209189] Key type big_key registered [ 3.220736] Key type encrypted registered [ 3.221706] ima: No TPM chip found, activating TPM-bypass! [ 3.222848] ima: Allocated hash algorithm: sha1 [ 3.223769] ima: No architecture policies found [ 3.225513] evm: Initialising EVM extended attributes: [ 3.227376] evm: security.selinux [ 3.229953] evm: security.ima [ 3.230893] evm: security.capability [ 3.231917] evm: HMAC attrs: 0x1 [ 3.234052] rtc_cmos 00:05: setting system clock to 2026-08-14 23:12:04 UTC (1786749124) [ 3.238976] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.241354] debug: unmapping init [mem 0xffffffff86d82000-0xffffffff87058fff] [ 3.249084] Write protecting the kernel read-only data: 28672k [ 3.251719] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 3.254485] debug: unmapping init [mem 0xffffffff85d14000-0xffffffff85dfffff] [ 3.287787] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.293953] systemd[1]: Detected virtualization kvm. [ 3.295752] systemd[1]: Detected architecture x86-64. [ 3.298313] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.322179] systemd[1]: No hostname configured. [ 3.324090] systemd[1]: Set hostname to . [ 3.326245] random: systemd: uninitialized urandom read (16 bytes read) [ 3.328058] systemd[1]: Initializing machine ID from random generator. [ 3.468668] random: systemd: uninitialized urandom read (16 bytes read) [ 3.472066] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.475729] random: systemd: uninitialized urandom read (16 bytes read) [ 3.479160] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.484136] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.163376] device-mapper: uevent: version 1.0.3 [ 4.165949] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.946243] random: fast init done [ 4.949356] virtio_net virtio0 ens2: renamed from eth0 [ 5.073335] scsi host0: ata_piix [ 5.108141] scsi host1: ata_piix [ 5.109412] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.111499] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.930840] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.739794] random: crng init done [ 9.741813] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.269629] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.396126] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.638973] SELinux: Disabled at runtime. [ 11.699687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.708254] systemd[1]: Detected virtualization kvm. [ 11.710200] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.263416] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.268605] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.274400] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.280889] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.284761] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.293952] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.299140] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ 12.353247] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.800025] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.160945] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.329934] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.394972] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.405874] EDAC sbridge: Ver: 1.1.2 [ 15.082580] Key type dns_resolver registered [ 15.397453] NFS: Registering the id_resolver key type [ 15.399597] Key type id_resolver registered [ 15.401297] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 32.847950] spl: loading out-of-tree module taints kernel. [ 35.480543] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 40.319878] Key type ._llcrypt registered [ 40.321747] Key type .llcrypt registered [ 40.378444] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 48.672365] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 50.337858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.353722] alg: No test for adler32 (adler32-zlib) [ 52.004238] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 53.271258] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 55.087276] Key type lgssc registered [ 57.329748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.937192] vdc: vdc1 vdc9 [ 77.437567] vde: vde1 vde9 [ 86.427339] vdf: vdf1 vdf9 [ 107.311832] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 117.368044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 118.855814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 119.268277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 119.464641] Lustre: lustre-MDT0000: new disk, initializing [ 120.082492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 120.155948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 124.955149] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.394221] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 136.684892] Lustre: lustre-OST0000: new disk, initializing [ 136.687373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 136.690306] Lustre: Skipped 1 previous similar message [ 136.770800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.349073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 137.358455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 137.477349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 143.238312] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.621433] Lustre: lustre-OST0001: new disk, initializing [ 157.627409] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 157.820187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.250709] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.049375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 165.081870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 165.309072] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 176.091905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.044967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 188.666650] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 194.738424] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 200.406248] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 202.753335] hrtimer: interrupt took 3031054 ns [ 203.300278] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 205.582961] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 207.226131] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Aug 14 19:15:27 EDT 2026 [ 226.812210] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 229.371591] Lustre: DEBUG MARKER: === replay-single: start setup 19:15:49 (1786749349) === [ 234.775928] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing check_config_client /mnt/lustre [ 251.639961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.212632] Lustre: 11310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 258.798575] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.202608] Lustre: DEBUG MARKER: === replay-single: finish setup 19:16:23 (1786749383) === [ 264.918544] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:16:25 (1786749385) [ 267.834163] LustreError: 11807:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 268.567241] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 270.500512] Lustre: Failing over lustre-MDT0000 [ 270.816589] Lustre: server umount lustre-MDT0000 complete [ 288.551583] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749393/real 1786749393] req@ffff9af9c3f77480 x1873542301931648/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786749409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 288.573581] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 288.582330] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.727196] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749398/real 1786749398] req@ffff9af9fdd81500 x1873542301932288/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 293.758381] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 297.887198] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749403/real 1786749403] req@ffff9af9fdd80000 x1873542301932672/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 297.893855] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c0aefb [ 297.941409] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 297.973500] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 298.336996] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (not set up) [ 298.541940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 299.497607] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 300.093534] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 300.207091] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 302.576570] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 303.519317] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749408/real 1786749408] req@ffff9af9d8465500 x1873542301933056/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 303.565379] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 311.773273] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 312.801217] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 313.328954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 321.370112] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:17:21 (1786749441) [ 323.038537] Lustre: Failing over lustre-OST0000 [ 323.049208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.056479] Lustre: Skipped 1 previous similar message [ 323.145210] Lustre: server umount lustre-OST0000 complete [ 329.059487] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.089920] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 333.283670] LustreError: 6692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.400236] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.423473] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 341.377292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.576720] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 343.042395] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 343.043088] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 349.005301] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 358.746972] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 360.397741] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 369.032669] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 19:18:09 (1786749489) [ 371.704829] LustreError: 14850:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 372.390277] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 374.444888] Lustre: Failing over lustre-MDT0000 [ 374.775382] Lustre: server umount lustre-MDT0000 complete [ 392.858356] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.051043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.210827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.079634] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749498/real 1786749498] req@ffff9af8c6ae6300 x1873542301963520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 394.142080] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 397.935477] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 398.324740] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 400.502881] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 400.510134] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 403.295232] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749508/real 1786749508] req@ffff9af9f44f2680 x1873542301964416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 403.313711] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 405.856744] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 410.973751] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 416.093920] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 421.213772] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 431.455847] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 431.474155] Lustre: Skipped 1 previous similar message [ 451.938569] Lustre: lustre-MDT0000: Denying connection for new client 9b004775-21a1-413b-88e6-32490c97441b (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:08 [ 451.966411] Lustre: Skipped 3 previous similar messages [ 460.500283] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 460.510432] Lustre: 15487:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 1a4544ff-6fd2-404d-8a28-c0bd738fe8c2@ [ 460.537468] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 460.602991] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 460.633032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:65) [ 460.639694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:65) [ 471.087662] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 19:19:51 (1786749591) [ 474.250216] LustreError: 16226:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 475.244847] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 477.024443] Lustre: Failing over lustre-MDT0000 [ 479.354131] Lustre: server umount lustre-MDT0000 complete [ 496.607224] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749601/real 1786749601] req@ffff9af8c7c27b80 x1873542301989888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 496.608183] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.609671] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.609678] Lustre: Skipped 1 previous similar message [ 496.635113] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 506.297961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.413523] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 512.653226] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 512.666301] Lustre: lustre-MDT0000: Denying connection for new client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 512.678258] Lustre: Skipped 1 previous similar message [ 521.584412] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 521.600673] Lustre: Skipped 1 previous similar message [ 572.500323] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 572.502602] Lustre: 16869:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9b004775-21a1-413b-88e6-32490c97441b@ [ 572.530601] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 572.609801] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 572.668870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:97) [ 572.671872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:97) [ 585.866684] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 19:21:45 (1786749705) [ 588.707226] LustreError: 17606:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 589.678598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 591.443730] Lustre: Failing over lustre-MDT0000 [ 591.952428] Lustre: server umount lustre-MDT0000 complete [ 608.736063] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749714/real 1786749714] req@ffff9af9dd234a80 x1873542302012800/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786749730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 608.786650] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 608.803766] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.759221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.771393] Lustre: Skipped 1 previous similar message [ 619.324475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.454218] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 620.264508] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 620.275814] Lustre: Skipped 1 previous similar message [ 620.408416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 620.612987] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 620.677869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:129) [ 620.681729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:129) [ 623.798461] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 632.556135] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 633.777904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 641.732746] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 19:22:41 (1786749761) [ 644.606168] LustreError: 19197:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 645.505544] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 647.885632] Lustre: Failing over lustre-MDT0000 [ 648.295253] Lustre: server umount lustre-MDT0000 complete [ 667.280807] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.540249] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.548832] Lustre: Skipped 2 previous similar messages [ 667.811923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 672.761944] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 672.771222] Lustre: Skipped 1 previous similar message [ 672.887938] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 676.719951] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 676.832116] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749782/real 1786749782] req@ffff9af9f4525f80 x1873542302029696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 676.882204] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 676.964213] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 677.009431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 677.014386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:161) [ 683.600942] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 685.413819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 694.078877] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 19:23:34 (1786749814) [ 697.102529] LustreError: 20780:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 698.050428] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 699.991945] Lustre: Failing over lustre-MDT0000 [ 700.446686] Lustre: server umount lustre-MDT0000 complete [ 718.753704] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.647237] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.556536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.561865] Lustre: Skipped 1 previous similar message [ 729.627724] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 734.411682] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 743.263390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 743.467317] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 743.560408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:193) [ 743.571657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 744.677655] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 744.690858] Lustre: Skipped 2 previous similar messages [ 749.454731] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 751.020371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 759.169550] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 19:24:39 (1786749879) [ 761.984699] LustreError: 22376:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 762.849804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 764.769114] Lustre: Failing over lustre-MDT0000 [ 765.191216] Lustre: server umount lustre-MDT0000 complete [ 781.792135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.793960] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.809550] Lustre: Skipped 2 previous similar messages [ 792.033395] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9fed38380 x1873542302064384/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 793.105031] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 795.487720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 795.701472] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 795.776337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 795.778583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:225) [ 799.158911] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 811.251369] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 812.499546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 823.148983] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 19:25:42 (1786749942) [ 825.468609] LustreError: 23972:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 826.151801] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 827.467893] Lustre: Failing over lustre-MDT0000 [ 827.806416] Lustre: server umount lustre-MDT0000 complete [ 844.256787] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786749949/real 1786749949] req@ffff9af9feaa1c00 x1873542302078592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786749965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 844.292195] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [ 846.465202] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.663313] LustreError: 24568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.676528] LustreError: 24568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 847.210699] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 848.357807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:257) [ 848.362695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:257) [ 852.441441] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 863.339798] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.138828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 874.752501] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 19:26:34 (1786749994) [ 875.879963] Lustre: *** cfs_fail_loc=13b, val=315*** [ 875.892770] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 875.894800] LustreError: 24572:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9fed38000 x1873542291511296/t38654705666(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:234/0 lens 392/456 e 0 to 0 dl 1786750014 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 880.350981] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 882.481563] Lustre: Failing over lustre-MDT0000 [ 882.557558] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 882.812267] Lustre: server umount lustre-MDT0000 complete [ 901.600483] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.619370] Lustre: Skipped 3 previous similar messages [ 911.717060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.725270] Lustre: Skipped 2 previous similar messages [ 911.969019] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 916.643974] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 922.467667] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 922.482764] Lustre: Skipped 1 previous similar message [ 922.549888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 922.552494] Lustre: Skipped 1 previous similar message [ 922.628186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:289) [ 922.631182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:289) [ 922.675020] Lustre: 26206:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9fed38a80 x1873542291511296/t38654705666(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:280/0 lens 392/456 e 0 to 0 dl 1786750060 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 925.793800] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 925.809448] Lustre: Skipped 4 previous similar messages [ 928.538246] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 930.467419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 941.089402] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 19:27:40 (1786750060) [ 944.743749] LustreError: 27190:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 944.748786] LustreError: 27190:0:(osd_handler.c:715:osd_ro()) Skipped 1 previous similar message [ 945.833611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 948.212800] Lustre: Failing over lustre-MDT0000 [ 948.910654] Lustre: server umount lustre-MDT0000 complete [ 968.159537] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.175690] LustreError: Skipped 1 previous similar message [ 978.400030] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af8c6203b80 x1873542302114432/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 979.344446] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 980.081911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:321) [ 980.082868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:321) [ 985.550866] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 996.591314] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 998.625976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1008.114500] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 19:28:48 (1786750128) [ 1012.555282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1013.634820] Lustre: *** cfs_fail_loc=114, val=0*** [ 1016.359408] Lustre: Failing over lustre-MDT0000 [ 1016.629863] Lustre: server umount lustre-MDT0000 complete [ 1035.546323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.567034] Lustre: Skipped 2 previous similar messages [ 1035.806885] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1036.264837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:353) [ 1036.269236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:353) [ 1039.636252] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1050.443308] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1052.601463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1062.458082] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 19:29:42 (1786750182) [ 1066.948682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1067.727422] Lustre: *** cfs_fail_loc=128, val=0*** [ 1070.656587] Lustre: Failing over lustre-MDT0000 [ 1071.002577] Lustre: server umount lustre-MDT0000 complete [ 1097.186759] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9d8efed80 x1873542302147712/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1097.581261] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (not set up) [ 1099.122368] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1099.149235] Lustre: Skipped 2 previous similar messages [ 1099.282517] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1099.298418] Lustre: Skipped 2 previous similar messages [ 1099.409331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:385) [ 1099.410065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:385) [ 1103.843619] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750208/real 1786750208] req@ffff9af9fdd81180 x1873542302147456/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786750224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1103.894086] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 42 previous similar messages [ 1104.662642] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1115.948672] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1117.358440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1125.061347] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 19:30:45 (1786750245) [ 1129.626290] LustreError: 32126:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1129.640169] LustreError: 32126:0:(osd_handler.c:715:osd_ro()) Skipped 2 previous similar messages [ 1131.096598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1134.965259] Lustre: Failing over lustre-MDT0000 [ 1135.696290] Lustre: server umount lustre-MDT0000 complete [ 1152.999134] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1153.013675] LustreError: Skipped 2 previous similar messages [ 1163.914567] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1163.927720] Lustre: Skipped 1 previous similar message [ 1165.680435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:391 to 0x240000400:417) [ 1165.680534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:391 to 0x280000400:417) [ 1169.147593] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1180.036902] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.773702] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1191.170289] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 19:31:51 (1786750311) [ 1195.178903] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1197.372808] Lustre: Failing over lustre-MDT0000 [ 1197.780693] Lustre: server umount lustre-MDT0000 complete [ 1224.179336] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c0f69f [ 1224.208846] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 1224.220781] Lustre: Skipped 9 previous similar messages [ 1224.650152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.656723] Lustre: Skipped 4 previous similar messages [ 1229.562827] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1239.328526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:423 to 0x240000400:449) [ 1239.332766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:423 to 0x280000400:449) [ 1245.120634] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.918565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1255.667195] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 19:32:56 (1786750376) [ 1259.582346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1267.780745] Lustre: Failing over lustre-MDT0000 [ 1268.149590] Lustre: server umount lustre-MDT0000 complete [ 1295.334527] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c12b81 [ 1296.054406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1296.067390] Lustre: Skipped 1 previous similar message [ 1301.829927] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1301.875870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:577) [ 1301.878543] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:577) [ 1312.992565] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1314.644890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1339.890391] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 19:34:19 (1786750459) [ 1343.526994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1345.384200] Lustre: Failing over lustre-MDT0000 [ 1345.720110] Lustre: server umount lustre-MDT0000 complete [ 1362.399424] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.403701] Lustre: Skipped 10 previous similar messages [ 1374.067759] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1374.083611] Lustre: Skipped 3 previous similar messages [ 1374.231163] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1374.235976] Lustre: Skipped 3 previous similar messages [ 1374.281530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:609) [ 1374.281955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:609) [ 1378.397959] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1387.953924] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.516701] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1400.197352] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 19:35:20 (1786750520) [ 1403.832521] LustreError: 38539:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1403.841094] LustreError: 38539:0:(osd_handler.c:715:osd_ro()) Skipped 3 previous similar messages [ 1405.037426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1407.635766] Lustre: Failing over lustre-MDT0000 [ 1408.077241] Lustre: server umount lustre-MDT0000 complete [ 1423.775110] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1423.790043] LustreError: Skipped 3 previous similar messages [ 1435.854113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:641) [ 1435.855040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:641) [ 1440.273260] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1451.894681] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1453.465591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1462.413533] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 19:36:22 (1786750582) [ 1466.267298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1468.619751] Lustre: Failing over lustre-MDT0000 [ 1469.029742] Lustre: server umount lustre-MDT0000 complete [ 1495.008099] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9c3f75f80 x1873542302315008/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1498.048616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:673) [ 1498.060547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:673) [ 1500.438369] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1509.640876] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.113890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1520.070825] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 19:37:19 (1786750639) [ 1524.133472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.174735] Lustre: Failing over lustre-MDT0000 [ 1526.496425] Lustre: server umount lustre-MDT0000 complete [ 1551.914338] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1555.645048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:705) [ 1555.646148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:705) [ 1564.056822] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.454621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1577.275259] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 19:38:16 (1786750696) [ 1581.624541] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1583.770637] Lustre: Failing over lustre-MDT0000 [ 1584.221528] Lustre: server umount lustre-MDT0000 complete [ 1614.383407] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1614.396453] Lustre: Skipped 4 previous similar messages [ 1620.374884] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1627.204119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:737) [ 1627.218422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:737) [ 1633.040643] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1634.436241] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1642.772719] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 19:39:23 (1786750763) [ 1647.075670] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1649.115858] Lustre: Failing over lustre-MDT0000 [ 1649.401133] Lustre: server umount lustre-MDT0000 complete [ 1670.987689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:769) [ 1670.991702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:769) [ 1673.673530] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1685.183521] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1686.436449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1696.097994] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 19:40:15 (1786750815) [ 1700.149427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1702.502122] Lustre: Failing over lustre-MDT0000 [ 1702.844983] Lustre: server umount lustre-MDT0000 complete [ 1720.799100] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750826/real 1786750826] req@ffff9af9ffce0700 x1873542302378880/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786750842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1720.818611] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 64 previous similar messages [ 1735.446633] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1746.004826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:801) [ 1746.009288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:801) [ 1751.650677] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.344761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1761.522248] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 19:41:21 (1786750881) [ 1765.609190] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1767.714941] Lustre: Failing over lustre-MDT0000 [ 1768.193945] Lustre: server umount lustre-MDT0000 complete [ 1794.495126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.498780] Lustre: Skipped 8 previous similar messages [ 1798.247714] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:833) [ 1798.247870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:833) [ 1798.810604] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1807.869467] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.869138] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1808.874194] Lustre: Skipped 19 previous similar messages [ 1809.258607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1816.967302] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 19:42:17 (1786750937) [ 1820.922361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1822.514314] Lustre: Failing over lustre-MDT0000 [ 1822.745498] Lustre: server umount lustre-MDT0000 complete [ 1850.216659] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (not set up) [ 1851.774553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:865) [ 1851.778799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:865) [ 1855.245985] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1866.247529] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.151929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1876.863728] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 19:43:16 (1786750996) [ 1881.721087] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1883.759309] Lustre: Failing over lustre-MDT0000 [ 1883.973107] Lustre: server umount lustre-MDT0000 complete [ 1903.335331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.371956] Lustre: Skipped 16 previous similar messages [ 1912.545209] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c1bc20 [ 1917.773201] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1927.008385] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1927.032097] Lustre: Skipped 8 previous similar messages [ 1927.199579] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1927.216691] Lustre: Skipped 8 previous similar messages [ 1927.278387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:867 to 0x240000400:897) [ 1927.303200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:897) [ 1933.732446] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.552664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1943.724487] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 19:44:23 (1786751063) [ 1946.443341] LustreError: 52837:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1946.448756] LustreError: 52837:0:(osd_handler.c:715:osd_ro()) Skipped 8 previous similar messages [ 1947.213666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1949.132292] Lustre: Failing over lustre-MDT0000 [ 1949.443655] Lustre: server umount lustre-MDT0000 complete [ 1966.559239] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.576109] LustreError: Skipped 8 previous similar messages [ 1975.841181] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9c3f77b80 x1873542302447616/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1979.548400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:929) [ 1979.549721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:899 to 0x240000400:929) [ 1983.387708] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1994.021476] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1995.642921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2004.410977] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 19:45:24 (1786751124) [ 2008.423851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2010.899917] Lustre: Failing over lustre-MDT0000 [ 2011.443143] Lustre: server umount lustre-MDT0000 complete [ 2037.218262] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c1c5f1 [ 2040.980174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:931 to 0x240000400:961) [ 2040.980514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:961) [ 2042.938258] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2052.939440] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.435565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2062.284267] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 19:46:22 (1786751182) [ 2066.085759] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2068.022723] Lustre: Failing over lustre-MDT0000 [ 2068.297264] Lustre: server umount lustre-MDT0000 complete [ 2089.795217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:963 to 0x280000400:993) [ 2089.795813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:931 to 0x240000400:993) [ 2092.537914] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2102.753070] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.889803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2113.394583] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 19:47:13 (1786751233) [ 2117.364210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2119.783512] Lustre: Failing over lustre-MDT0000 [ 2120.219601] Lustre: server umount lustre-MDT0000 complete [ 2139.111741] LustreError: 58204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.852624] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2139.855920] Lustre: Skipped 8 previous similar messages [ 2140.258545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:995 to 0x240000400:1025) [ 2140.268287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:995 to 0x280000400:1025) [ 2144.715917] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2154.615566] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2156.230925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2164.371395] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 19:48:04 (1786751284) [ 2167.753480] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2169.687276] Lustre: Failing over lustre-MDT0000 [ 2170.154702] Lustre: server umount lustre-MDT0000 complete [ 2195.935869] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9f769aa00 x1873542302510336/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2197.549478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1057) [ 2197.551040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1027 to 0x240000400:1057) [ 2203.158203] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2213.231873] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.244033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2225.636051] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:49:05 (1786751345) [ 2229.112082] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2231.687244] Lustre: Failing over lustre-MDT0000 [ 2232.103551] Lustre: server umount lustre-MDT0000 complete [ 2261.986091] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c1daea [ 2266.070908] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2274.414632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1089) [ 2274.415561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1059 to 0x240000400:1089) [ 2280.466571] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2282.152423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2290.105628] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 19:50:10 (1786751410) [ 2294.949867] Lustre: 62420:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2513f734-913c-483a-be19-9bc2215f95a9 at adminstrative request [ 2300.902642] Lustre: Failing over lustre-MDT0000 [ 2301.183791] Lustre: server umount lustre-MDT0000 complete [ 2323.871159] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786751429/real 1786751429] req@ffff9af9d196e300 x1873542302543872/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786751445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2323.909227] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 89 previous similar messages [ 2330.015496] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9c5e65180 x1873542302545408/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2332.272432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1121) [ 2332.274513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1091 to 0x240000400:1121) [ 2335.312824] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2345.492431] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2347.156718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2353.193799] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2368.091282] Lustre: DEBUG MARKER: before 4096, after 4096 [ 2374.845378] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 19:51:35 (1786751495) [ 2375.896258] Lustre: 64506:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2513f734-913c-483a-be19-9bc2215f95a9 at adminstrative request [ 2383.904705] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:51:44 (1786751504) [ 2387.308733] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2389.283887] Lustre: Failing over lustre-MDT0000 [ 2391.008450] LustreError: 63030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.033067] LustreError: 63030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2391.649757] Lustre: server umount lustre-MDT0000 complete [ 2417.508829] LustreError: 65449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.918311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.924625] Lustre: Skipped 9 previous similar messages [ 2419.304501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1124 to 0x240000400:1153) [ 2419.307331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1123 to 0x280000400:1153) [ 2422.863461] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2431.471106] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2431.483421] Lustre: Skipped 22 previous similar messages [ 2434.057913] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2435.454975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2443.585596] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:52:43 (1786751563) [ 2447.643822] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2449.674133] Lustre: Failing over lustre-MDT0000 [ 2449.950239] Lustre: server umount lustre-MDT0000 complete [ 2481.086991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1155 to 0x240000400:1185) [ 2481.094137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1155 to 0x280000400:1185) [ 2483.625186] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2492.287443] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.826179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2501.810877] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:53:41 (1786751621) [ 2505.627438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2507.284617] Lustre: Failing over lustre-MDT0000 [ 2507.501484] Lustre: server umount lustre-MDT0000 complete [ 2524.896401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.914582] Lustre: Skipped 19 previous similar messages [ 2536.295261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2536.304884] Lustre: Skipped 9 previous similar messages [ 2536.500343] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2536.508350] Lustre: Skipped 9 previous similar messages [ 2536.570445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1187 to 0x280000400:1217) [ 2536.576696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1187 to 0x240000400:1217) [ 2540.524314] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2549.705491] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2551.019657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2558.782236] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 19:54:39 (1786751679) [ 2561.833902] LustreError: 69591:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2561.842435] LustreError: 69591:0:(osd_handler.c:715:osd_ro()) Skipped 8 previous similar messages [ 2562.741639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2564.675712] Lustre: Failing over lustre-MDT0000 [ 2565.114394] Lustre: server umount lustre-MDT0000 complete [ 2580.837336] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.853142] LustreError: Skipped 9 previous similar messages [ 2592.774055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1219 to 0x240000400:1249) [ 2592.779103] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1249) [ 2596.211089] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2605.814932] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.690871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2616.415408] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:55:36 (1786751736) [ 2620.757322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2622.643659] Lustre: Failing over lustre-MDT0000 [ 2623.099408] Lustre: server umount lustre-MDT0000 complete [ 2646.916136] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2649.088485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1251 to 0x280000400:1281) [ 2649.089079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1219 to 0x240000400:1281) [ 2657.123779] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2658.864169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2667.440444] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:56:27 (1786751787) [ 2671.324545] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2673.631874] Lustre: Failing over lustre-MDT0000 [ 2673.959066] Lustre: server umount lustre-MDT0000 complete [ 2692.578134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2698.364465] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2701.438612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1283 to 0x240000400:1313) [ 2701.448139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1283 to 0x280000400:1313) [ 2708.266588] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.721161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2718.794729] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:57:18 (1786751838) [ 2723.001054] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2725.023369] Lustre: Failing over lustre-MDT0000 [ 2725.563490] Lustre: server umount lustre-MDT0000 complete [ 2743.965281] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2743.976962] Lustre: Skipped 9 previous similar messages [ 2748.656885] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2753.546980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1315 to 0x280000400:1345) [ 2753.551614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1315 to 0x240000400:1345) [ 2760.025520] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2761.683111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2771.930698] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:58:12 (1786751892) [ 2776.200313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2778.136346] Lustre: Failing over lustre-MDT0000 [ 2778.561743] Lustre: server umount lustre-MDT0000 complete [ 2805.605302] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (not set up) [ 2807.687502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1347 to 0x240000400:1377) [ 2807.688970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1347 to 0x280000400:1377) [ 2811.228280] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2822.938175] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.556108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2833.935385] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:59:14 (1786751954) [ 2838.416581] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2840.621053] Lustre: Failing over lustre-MDT0000 [ 2840.946947] Lustre: server umount lustre-MDT0000 complete [ 2865.516370] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2868.296847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1379 to 0x240000400:1409) [ 2868.298056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1379 to 0x280000400:1409) [ 2875.453178] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2877.114377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2887.003968] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:00:06 (1786752006) [ 2891.767664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2894.103752] Lustre: Failing over lustre-MDT0000 [ 2894.324314] Lustre: server umount lustre-MDT0000 complete [ 2921.965188] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c2130d [ 2924.601193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1411 to 0x240000400:1441) [ 2924.601371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1411 to 0x280000400:1441) [ 2927.080465] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752032/real 1786752032] req@ffff9af9d1916d80 x1873542302711040/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786752048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2927.129297] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 68 previous similar messages [ 2927.305155] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2941.039244] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2943.686207] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2952.589912] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:01:12 (1786752072) [ 2956.719883] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2958.903593] Lustre: Failing over lustre-MDT0000 [ 2959.260891] Lustre: server umount lustre-MDT0000 complete [ 2983.202567] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2986.127900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1443 to 0x280000400:1473) [ 2986.133457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1443 to 0x240000400:1473) [ 2993.126894] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2995.012733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3003.196940] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:02:03 (1786752123) [ 3004.385062] Lustre: 82214:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2513f734-913c-483a-be19-9bc2215f95a9 at adminstrative request [ 3013.704594] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:02:14 (1786752134) [ 3016.316708] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3018.473268] Lustre: Failing over lustre-MDT0000 [ 3019.744815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.021486] Lustre: server umount lustre-MDT0000 complete [ 3033.603621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.608805] Lustre: Skipped 10 previous similar messages [ 3033.778590] Lustre: lustre-MDT0000: Aborting client recovery [ 3033.782734] LustreError: 83131:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3033.795578] Lustre: 83177:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3033.806389] Lustre: 83177:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3033.826641] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3033.920896] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3034.038835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1479 to 0x280000400:1505) [ 3034.041738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1480 to 0x240000400:1505) [ 3039.459870] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3040.749432] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3040.758856] Lustre: Skipped 22 previous similar messages [ 3052.593791] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:02:52 (1786752172) [ 3056.742975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3058.986233] Lustre: Failing over lustre-MDT0000 [ 3059.469563] Lustre: server umount lustre-MDT0000 complete [ 3068.427646] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3068.494739] Lustre: lustre-MDT0000: Aborting client recovery [ 3068.499696] LustreError: 84497:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3068.510179] Lustre: 84544:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3068.528089] Lustre: 84544:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3068.538063] Lustre: 84544:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3068.549993] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3068.626793] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3068.774070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1516 to 0x240000400:1537) [ 3068.777417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1516 to 0x280000400:1537) [ 3074.740818] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3080.254349] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3087.045681] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:03:27 (1786752207) [ 3090.803931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3093.002231] Lustre: Failing over lustre-MDT0000 [ 3093.346472] LustreError: 84507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.627519] Lustre: server umount lustre-MDT0000 complete [ 3103.303214] Lustre: lustre-MDT0000: Aborting client recovery [ 3103.304832] LustreError: 85868:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3103.309771] Lustre: 85915:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3103.322309] Lustre: 85915:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3103.327403] Lustre: 85915:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3103.337906] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3103.407070] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3103.647648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1569) [ 3103.652741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1544 to 0x280000400:1569) [ 3107.155178] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3123.338749] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:04:03 (1786752243) [ 3124.508072] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3124.519271] LustreError: 85881:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9feaa2d80 x1873542292509440/t201863462916(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:211/0 lens 512/456 e 0 to 0 dl 1786752256 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3128.387759] Lustre: Failing over lustre-MDT0000 [ 3128.801921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.822622] Lustre: Skipped 23 previous similar messages [ 3128.838156] LustreError: 85879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.855911] LustreError: 85879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3128.993619] Lustre: server umount lustre-MDT0000 complete [ 3138.198910] Lustre: lustre-MDT0000: Aborting client recovery [ 3138.201928] LustreError: 87083:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3138.207460] Lustre: 87130:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3138.212905] Lustre: 87130:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3138.220484] Lustre: 87130:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3138.228901] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3138.269050] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3138.355886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1601) [ 3138.357410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1571 to 0x280000400:1601) [ 3143.254698] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3157.340504] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3158.880065] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:04:39 (1786752279) [ 3161.866484] LustreError: 87885:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3161.870575] LustreError: 87885:0:(osd_handler.c:715:osd_ro()) Skipped 10 previous similar messages [ 3162.794657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3165.587871] Lustre: Failing over lustre-MDT0000 [ 3166.042827] Lustre: server umount lustre-MDT0000 complete [ 3175.273077] LustreError: 88565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.874747] Lustre: lustre-MDT0000: Aborting client recovery [ 3175.878207] LustreError: 88556:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3175.884309] Lustre: 88604:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3175.890157] Lustre: 88604:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3175.895511] Lustre: 88604:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3175.903703] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3175.935899] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3176.041105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1543 to 0x240000400:1633) [ 3176.046898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1571 to 0x280000400:1633) [ 3181.076762] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.079814] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:05:16 (1786752316) [ 3226.361271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3228.752388] Lustre: Failing over lustre-MDT0000 [ 3229.194964] Lustre: server umount lustre-MDT0000 complete [ 3248.659513] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3248.676750] LustreError: Skipped 12 previous similar messages [ 3254.973058] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3258.212732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3258.228672] Lustre: Skipped 8 previous similar messages [ 3258.317621] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3258.332528] Lustre: Skipped 8 previous similar messages [ 3258.380836] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049) [ 3258.382235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049) [ 3265.085939] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3266.576784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3293.001294] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:06:53 (1786752413) [ 3313.102384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3323.785797] Lustre: Failing over lustre-MDT0000 [ 3324.153861] Lustre: server umount lustre-MDT0000 complete [ 3352.171579] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3352.192482] Lustre: Skipped 20 previous similar messages [ 3357.775742] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3369.534201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465) [ 3369.540833] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465) [ 3376.863610] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3378.534627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3400.112346] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:08:40 (1786752520) [ 3402.615987] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3403.595319] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3403.602597] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3410.443757] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:08:50 (1786752530) [ 3434.467774] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3446.283255] Lustre: Failing over lustre-OST0000 [ 3446.396405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3446.408348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.418294] LustreError: Skipped 4 previous similar messages [ 3446.443110] Lustre: Skipped 1 previous similar message [ 3446.687706] Lustre: server umount lustre-OST0000 complete [ 3447.687892] LustreError: 36583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.767554] LustreError: 14851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.792960] LustreError: 14851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3463.006478] LustreError: 36589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3463.040307] LustreError: 36589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3475.175466] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3534.564988] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:10:54 (1786752654) [ 3538.826481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3541.409795] Lustre: Failing over lustre-MDT0000 [ 3541.699971] Lustre: server umount lustre-MDT0000 complete [ 3559.903474] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752665/real 1786752665] req@ffff9af9d8ecf480 x1873542303312256/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786752681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3559.938762] Lustre: 3294:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 36 previous similar messages [ 3570.080753] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af8cc8c5180 x1873542303314048/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3575.243619] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3584.978212] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3584.979073] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881) [ 3591.725444] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3594.504620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3601.377777] LustreError: 96585:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3601.381882] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3602.405199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913) [ 3614.622639] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 20:12:14 (1786752734) [ 3620.407646] LustreError: 96562:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3625.439302] LustreError: 96562:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3625.450948] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3625.504226] LustreError: 6690:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 waking [ 3627.825581] LustreError: 96560:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3633.121172] LustreError: 96560:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3633.140137] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3635.470837] LustreError: 96562:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3640.799167] LustreError: 96562:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3640.816667] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3642.716731] LustreError: 97780:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3647.968584] LustreError: 97780:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3650.208253] LustreError: 97215:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3655.647182] LustreError: 97215:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3655.663445] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3655.678691] Lustre: Skipped 1 previous similar message [ 3658.410607] LustreError: 96560:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3663.839660] LustreError: 96560:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3679.199432] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3679.206917] Lustre: Skipped 2 previous similar messages [ 3681.495387] LustreError: 96563:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3681.504615] LustreError: 96563:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3686.879227] LustreError: 96563:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3686.900936] LustreError: 96563:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3704.014178] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 20:13:44 (1786752824) [ 3705.991705] LustreError: 97215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3715.424480] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3718.430310] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3720.477482] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3725.682069] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3730.781910] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3730.789991] Lustre: Skipped 1 previous similar message [ 3741.024135] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3741.030787] Lustre: Skipped 1 previous similar message [ 3746.071242] LustreError: 97215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3746.082781] Lustre: 97215:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9af8c8fb0700 x1873542295150464/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786752847 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3746.143323] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3746.151270] Lustre: Skipped 3 previous similar messages [ 3746.156111] LustreError: 97780:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3770.720721] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3786.233128] LustreError: 97780:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3786.246543] Lustre: 97780:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9af8c8c77480 x1873542295153792/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786752887 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3790.692744] LustreError: 96560:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3816.287838] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3816.304939] Lustre: Skipped 3 previous similar messages [ 3830.719117] LustreError: 96560:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3830.737330] Lustre: 96560:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff9af9f336a300 x1873542295155840/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786752931 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3831.646074] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 3831.661356] Lustre: Skipped 1 previous similar message [ 3831.668581] LustreError: 97215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3871.735157] LustreError: 97215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3871.751035] Lustre: 97215:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff9af8c65d6680 x1873542295157760/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786752972 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3872.610408] LustreError: 96563:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3898.219890] Lustre: lustre-MDT0000: Export ffff9af9c4ec1000 already connecting from 192.168.202.49@tcp [ 3898.230911] Lustre: Skipped 7 previous similar messages [ 3912.692287] LustreError: 96563:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3912.708350] Lustre: 96563:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9af9e6bf0e00 x1873542295159680/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786753013 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3913.580274] LustreError: 96560:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3937.097146] LustreError: 96560:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout interrupted [ 3937.110654] Lustre: 96560:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/4s); client may timeout req@ffff9af9f7698380 x1873542295161472/t0(0) o38->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1786753054 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3942.229383] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 20:17:42 (1786753062) [ 3945.386940] LustreError: 100433:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3945.395885] LustreError: 100433:0:(osd_handler.c:715:osd_ro()) Skipped 4 previous similar messages [ 3946.297687] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3950.604064] Lustre: Failing over lustre-MDT0000 [ 3950.949486] Lustre: server umount lustre-MDT0000 complete [ 3960.865836] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.875562] LustreError: Skipped 2 previous similar messages [ 3961.355573] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.358148] Lustre: *** cfs_fail_loc=712, val=0*** [ 3961.364353] Lustre: Skipped 12 previous similar messages [ 3961.371525] LustreError: 36583:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9af9fed91180 x1873542303408384/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3961.488770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.493436] Lustre: Skipped 8 previous similar messages [ 3961.577774] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3961.579159] Lustre: lustre-MDT0000: Aborting client recovery [ 3961.584176] Lustre: Skipped 2 previous similar messages [ 3961.584626] LustreError: 101111:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3961.588930] Lustre: 101158:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3961.591899] Lustre: 101158:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3961.594358] Lustre: 101158:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2513f734-913c-483a-be19-9bc2215f95a9@ [ 3961.599896] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3961.645295] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3961.759060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913) [ 3961.766652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945) [ 3966.447305] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3966.458761] Lustre: Skipped 18 previous similar messages [ 3967.672281] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3978.006455] Lustre: Failing over lustre-MDT0000 [ 3978.451663] Lustre: server umount lustre-MDT0000 complete [ 4002.520431] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4006.818150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4006.827356] Lustre: Skipped 3 previous similar messages [ 4006.914947] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4006.929542] Lustre: Skipped 3 previous similar messages [ 4006.985120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945) [ 4006.999870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977) [ 4013.313953] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4015.147552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4025.458783] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 20:19:05 (1786753145) [ 4025.639262] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 4025.658332] Lustre: Skipped 2 previous similar messages [ 4034.796800] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 20:19:14 (1786753154) [ 4035.867323] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4035.878906] LustreError: 102069:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9f3d5b480 x1873542295217152/t0(0) o700->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:368/0 lens 264/248 e 0 to 0 dl 1786753168 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4056.357499] Lustre: Failing over lustre-MDT0000 [ 4056.743275] Lustre: server umount lustre-MDT0000 complete [ 4086.782037] LustreError: 103769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.816052] LustreError: 103769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4091.769971] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4098.492795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977) [ 4098.492876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009) [ 4105.867523] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4107.472602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4118.918185] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:20:39 (1786753239) [ 4121.683217] Lustre: Failing over lustre-OST0000 [ 4121.769868] Lustre: server umount lustre-OST0000 complete [ 4123.106053] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.122556] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4150.844123] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4161.220657] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4162.954921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4236.987650] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:22:36 (1786753356) [ 4241.123694] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4243.653598] Lustre: Failing over lustre-MDT0000 [ 4243.925652] Lustre: server umount lustre-MDT0000 complete [ 4261.346212] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753366/real 1786753366] req@ffff9af9dd172680 x1873542303487360/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786753382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4261.391289] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 30 previous similar messages [ 4271.591829] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9ff099180 x1873542303489408/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4273.017483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3030 to 0x240000400:3073) [ 4273.026872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2998 to 0x280000400:3041) [ 4277.224698] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4344.924832] Lustre: *** cfs_fail_loc=216, val=0*** [ 4352.722054] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 20:24:32 (1786753472) [ 4355.578216] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4355.586966] Lustre: Skipped 2 previous similar messages [ 4369.339804] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 20:24:49 (1786753489) [ 4373.278271] Lustre: Failing over lustre-MDT0000 [ 4373.482653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.489662] Lustre: Skipped 1 previous similar message [ 4373.637929] Lustre: server umount lustre-MDT0000 complete [ 4404.128147] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9f3d58e00 x1873542303524864/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4409.345901] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4415.867908] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4415.872936] LustreError: 108570:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9ff782300 x1873542295337856/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:748/0 lens 328/344 e 0 to 0 dl 1786753548 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4432.221630] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 1:24 [ 4432.372929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3053 to 0x280000400:3073) [ 4432.376341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3084 to 0x240000400:3105) [ 4438.984679] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4440.495485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4452.110730] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 20:26:11 (1786753571) [ 4454.658783] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4459.231621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4461.268733] Lustre: Failing over lustre-MDT0000 [ 4461.722607] Lustre: server umount lustre-MDT0000 complete [ 4496.960592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3084 to 0x240000400:3137) [ 4496.975768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3105) [ 4497.467774] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4508.076368] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4510.192704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4519.668331] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 20:27:19 (1786753639) [ 4520.874500] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4527.340629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4529.650420] Lustre: Failing over lustre-MDT0000 [ 4529.994245] Lustre: server umount lustre-MDT0000 complete [ 4562.035054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3137) [ 4562.035994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3169) [ 4562.770899] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4572.705954] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4572.721062] Lustre: Skipped 14 previous similar messages [ 4574.841137] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4576.859880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4588.467460] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 20:28:28 (1786753708) [ 4589.913709] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4594.365024] LustreError: 113075:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4594.370458] LustreError: 113075:0:(osd_handler.c:715:osd_ro()) Skipped 3 previous similar messages [ 4595.289684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4597.215846] Lustre: Failing over lustre-MDT0000 [ 4597.679858] Lustre: server umount lustre-MDT0000 complete [ 4614.623181] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.626649] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.629989] Lustre: Skipped 17 previous similar messages [ 4614.644079] LustreError: Skipped 6 previous similar messages [ 4623.840489] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9ff58ca80 x1873542303580416/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4624.580796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.587360] Lustre: Skipped 7 previous similar messages [ 4624.664421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4624.674794] Lustre: Skipped 9 previous similar messages [ 4628.792087] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4631.009442] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4631.020933] Lustre: Skipped 6 previous similar messages [ 4631.050984] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4631.057147] Lustre: Skipped 6 previous similar messages [ 4631.107028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3169) [ 4631.107840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3171 to 0x240000400:3201) [ 4641.280927] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 20:29:21 (1786753761) [ 4643.495833] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4643.505795] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4643.514141] LustreError: 113692:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9fed91500 x1873542295386112/t257698037777(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:220/0 lens 392/456 e 0 to 0 dl 1786753775 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4646.649830] Lustre: Failing over lustre-MDT0000 [ 4646.763373] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 4646.968769] Lustre: server umount lustre-MDT0000 complete [ 4667.369044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3203 to 0x240000400:3233) [ 4667.370335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3075 to 0x280000400:3201) [ 4667.422924] Lustre: 114987:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9ff58b480 x1873542295386112/t257698037777(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:244/0 lens 392/456 e 0 to 0 dl 1786753799 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4670.550702] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4680.944766] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4682.861986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4691.866508] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 20:30:12 (1786753812) [ 4692.750801] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4692.752899] LustreError: 114984:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9ff58e300 x1873542295399424/t261993005072(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:270/0 lens 504/448 e 0 to 0 dl 1786753825 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4698.126547] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4700.111698] Lustre: Failing over lustre-MDT0000 [ 4700.483118] Lustre: server umount lustre-MDT0000 complete [ 4733.957197] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4734.011108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3233) [ 4734.011279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3203 to 0x240000400:3265) [ 4734.055424] Lustre: 116687:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af8c8ce6680 x1873542295399424/t261993005072(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:311/0 lens 504/2880 e 0 to 0 dl 1786753866 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4744.262635] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4746.021525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4755.012748] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 20:31:14 (1786753874) [ 4755.955987] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4755.960230] LustreError: 116686:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9f769f800 x1873542295414272/t266287972368(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:333/0 lens 504/448 e 0 to 0 dl 1786753888 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4757.880872] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4761.218922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4763.169977] Lustre: Failing over lustre-MDT0000 [ 4763.494790] Lustre: server umount lustre-MDT0000 complete [ 4789.856943] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4797.059982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3265) [ 4797.060378] Lustre: 118380:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9dd171c00 x1873542295414528/t266287972369(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:374/0 lens 392/456 e 0 to 0 dl 1786753929 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4797.064157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3267 to 0x240000400:3297) [ 4797.118901] Lustre: 118380:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4807.500740] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 20:32:07 (1786753927) [ 4808.597156] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4808.600390] Lustre: Skipped 1 previous similar message [ 4808.605501] LustreError: 118783:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af8c6ae5500 x1873542295426688/t270582939664(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:385/0 lens 504/448 e 0 to 0 dl 1786753940 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4808.623316] LustreError: 118783:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 4810.870720] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4810.886041] Lustre: Skipped 1 previous similar message [ 4817.074509] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4819.663694] Lustre: Failing over lustre-MDT0000 [ 4819.938654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.941411] Lustre: Skipped 1 previous similar message [ 4820.009763] Lustre: server umount lustre-MDT0000 complete [ 4845.604151] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4849.669567] Lustre: 119921:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9f3d58380 x1873542295426688/t270582939664(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:426/0 lens 504/2880 e 0 to 0 dl 1786753981 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4849.680122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3299 to 0x240000400:3329) [ 4849.707146] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3297) [ 4858.572783] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 20:32:58 (1786753978) [ 4859.663833] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4861.767242] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4861.772689] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4861.775669] LustreError: 119923:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9f3d59f80 x1873542295438976/t274877906960(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:439/0 lens 392/456 e 0 to 0 dl 1786753994 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4866.869635] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4868.980462] Lustre: Failing over lustre-MDT0000 [ 4869.308922] Lustre: server umount lustre-MDT0000 complete [ 4887.775286] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753993/real 1786753993] req@ffff9af9fe2adf80 x1873542303656704/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786754009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4887.801841] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 59 previous similar messages [ 4898.020690] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49742d9d97c6cacc [ 4901.812594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3361) [ 4901.815216] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3329) [ 4901.834245] Lustre: 121385:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9c5e65880 x1873542295438976/t274877906960(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:479/0 lens 392/456 e 0 to 0 dl 1786754034 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4903.115796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4915.908474] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 20:33:56 (1786754036) [ 4917.133205] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4917.143473] LustreError: 121476:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af8c5c67800 x1873542295449856/t279172874255(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:494/0 lens 664/608 e 0 to 0 dl 1786754049 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4932.447270] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnecting [ 4932.465756] Lustre: Skipped 1 previous similar message [ 4932.488694] Lustre: 121476:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9ff58ad80 x1873542295449856/t279172874255(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:509/0 lens 664/3488 e 0 to 0 dl 1786754064 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4939.797095] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 20:34:19 (1786754059) [ 4943.278359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4945.682786] Lustre: Failing over lustre-MDT0000 [ 4945.890661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4945.899204] Lustre: Skipped 1 previous similar message [ 4946.003290] Lustre: server umount lustre-MDT0000 complete [ 4970.393811] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4973.607496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3393) [ 4973.610083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3361) [ 4980.737709] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4983.277903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5004.295296] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 20:35:24 (1786754124) [ 5009.231966] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5011.522235] Lustre: Failing over lustre-MDT0000 [ 5011.974351] Lustre: server umount lustre-MDT0000 complete [ 5039.584033] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af8c8c74000 x1873542303696896/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5041.303068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3395 to 0x240000400:3425) [ 5041.303153] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3393) [ 5046.240961] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5057.211529] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.928779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5064.156857] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5076.301848] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 20:36:36 (1786754196) [ 5114.773680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5116.653988] Lustre: Failing over lustre-MDT0000 [ 5117.098594] Lustre: server umount lustre-MDT0000 complete [ 5140.665275] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5143.832033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4673) [ 5143.832831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4705) [ 5150.849547] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.523285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5207.260724] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 20:38:47 (1786754327) [ 5211.254815] LustreError: 128166:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 5211.270481] LustreError: 128166:0:(osd_handler.c:715:osd_ro()) Skipped 7 previous similar messages [ 5212.150087] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5213.984804] Lustre: Failing over lustre-MDT0000 [ 5214.427260] Lustre: server umount lustre-MDT0000 complete [ 5232.984467] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.996925] LustreError: Skipped 8 previous similar messages [ 5233.119957] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.139908] Lustre: Skipped 16 previous similar messages [ 5233.157222] LustreError: 128774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5233.187262] LustreError: 128774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5233.552462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.556298] Lustre: Skipped 8 previous similar messages [ 5233.620416] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5233.634042] Lustre: Skipped 8 previous similar messages [ 5234.616512] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5234.624119] Lustre: Skipped 8 previous similar messages [ 5237.913117] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5237.925804] Lustre: Skipped 8 previous similar messages [ 5237.989183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4737) [ 5237.996218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4675 to 0x280000400:4705) [ 5238.692634] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5238.760868] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5238.770932] Lustre: Skipped 20 previous similar messages [ 5249.158622] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5251.017195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5261.628975] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 5263.311690] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5271.938486] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 20:39:51 (1786754391) [ 5278.772667] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5297.496152] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5297.499162] Lustre: Skipped 1 previous similar message [ 5297.505525] LustreError: 128773:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af9d1545050 x1873542298232832/t296352743435(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:119/0 lens 66040/440 e 0 to 0 dl 1786754429 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5313.920404] Lustre: 128774:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af8c9110000 x1873542298232832/t296352743435(0) o36->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:136/0 lens 66040/440 e 0 to 0 dl 1786754446 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5323.482653] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5325.190408] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 20:40:45 (1786754445) [ 5334.663839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5337.708827] Lustre: Failing over lustre-MDT0000 [ 5338.213554] Lustre: server umount lustre-MDT0000 complete [ 5366.755305] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9ed74a680 x1873542304436096/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5373.165395] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5381.361897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4838 to 0x240000400:4865) [ 5381.363442] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4807 to 0x280000400:4833) [ 5387.080607] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5388.751559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5398.251961] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 20:41:58 (1786754518) [ 5419.878480] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5435.270590] Lustre: Failing over lustre-OST0000 [ 5435.338132] Lustre: server umount lustre-OST0000 complete [ 5437.921259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5437.937769] LustreError: Skipped 3 previous similar messages [ 5437.954500] LustreError: 36586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5437.965927] LustreError: 36586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5460.720404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5477.964743] Lustre: Failing over lustre-OST0000 [ 5478.007783] LustreError: 133806:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5478.023133] Lustre: 133243:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5478.031549] Lustre: 133243:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 5478.051029] Lustre: 133243:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@ffff9af8ce16bb80 x1873542304480640/t0(17179870643) o6->lustre-MDT0000-mdtlov_UUID@0@lo:302/0 lens 544/0 e 2 to 0 dl 1786754612 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5478.083342] LustreError: 133243:0:(ofd_obd.c:1324:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [ 5478.083924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -19 [ 5478.116430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5478.254328] Lustre: server umount lustre-OST0000 complete [ 5483.509518] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.554589] LustreError: 6691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5506.482595] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5510.109550] LustreError: 3291:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@ffff9af8cdb42a00 x1873542304480640/t17179870643(17179870643) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 3 to 0 dl 1786754650 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5518.283628] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5520.636378] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5562.871394] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 20:44:42 (1786754682) [ 5565.691927] Lustre: Failing over lustre-MDT0000 [ 5566.110488] Lustre: server umount lustre-MDT0000 complete [ 5583.327856] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786754688/real 1786754688] req@ffff9af9fe380a80 x1873542304617088/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786754704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5583.365604] Lustre: 3295:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 42 previous similar messages [ 5595.694842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5281) [ 5595.701167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5249) [ 5599.189894] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5614.008601] Lustre: Failing over lustre-MDT0000 [ 5614.391386] Lustre: server umount lustre-MDT0000 complete [ 5644.127711] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9fd76c000 x1873542304632832/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5649.373488] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5658.107436] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5281) [ 5658.107441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5313) [ 5664.742106] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5666.555470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5676.461610] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 20:46:36 (1786754796) [ 5691.379924] Lustre: Failing over lustre-OST0000 [ 5691.501735] Lustre: server umount lustre-OST0000 complete [ 5693.806150] LustreError: 36591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.821229] LustreError: 36591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5694.434809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.900297] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5728.769498] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5730.995183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5741.264816] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 20:47:41 (1786754861) [ 5743.497105] Lustre: Failing over lustre-MDT0000 [ 5743.909533] Lustre: server umount lustre-MDT0000 complete [ 5756.231726] Lustre: *** cfs_fail_loc=605, val=0*** [ 5756.243989] LustreError: 139838:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc10ed640 failed: rc = -95 [ 5756.266685] LustreError: 139838:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5756.290634] LustreError: 139838:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 5756.308694] LustreError: 139838:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5756.316637] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5756.325829] LustreError: 139838:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5756.422580] Lustre: server umount lustre-MDT0000 complete [ 5756.429060] LustreError: 139838:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5777.499548] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5786.159262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5283 to 0x280000400:5313) [ 5786.162278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5315 to 0x240000400:5345) [ 5787.589065] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 20:48:27 (1786754907) [ 5792.650041] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5796.895677] Lustre: Failing over lustre-MDT0000 [ 5797.261552] Lustre: server umount lustre-MDT0000 complete [ 5826.943343] Lustre: *** cfs_fail_loc=707, val=0*** [ 5829.892138] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5843.410822] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 5844.020175] Lustre: lustre-MDT0000: Recovery over after 0:18, of 1 clients 1 recovered and 0 were evicted. [ 5844.027778] Lustre: Skipped 7 previous similar messages [ 5844.075685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5327 to 0x280000400:5345) [ 5844.075685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5358 to 0x240000400:5377) [ 5850.841647] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.615232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5863.700354] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 20:49:43 (1786754983) [ 5895.371886] LustreError: 142314:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af8cc8c7800 x1873542299111040/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:717/0 lens 664/0 e 0 to 0 dl 1786755027 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5895.391891] LustreError: 142314:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5906.447936] LustreError: 142314:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5906.485239] LustreError: 141541:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af8c6ae4000 x1873542299112064/t0(0) o35->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:734/0 lens 392/0 e 0 to 0 dl 1786755044 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5909.295343] LustreError: 141538:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af8c903ca80 x1873542299118336/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:738/0 lens 576/0 e 0 to 0 dl 1786755048 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 5909.331183] LustreError: 141538:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 5915.694314] LustreError: 36581:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af9fdd80700 x1873542304707200/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:737/0 lens 544/0 e 0 to 0 dl 1786755047 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 5915.736223] LustreError: 36581:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 28 previous similar messages [ 5926.064985] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 20:50:46 (1786755046) [ 5956.731967] LustreError: 35572:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5967.831261] LustreError: 35572:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5978.809503] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 20:51:38 (1786755098) [ 6010.540587] LustreError: 141540:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af8c10b2d80 x1873542299136512/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:77/0 lens 576/0 e 0 to 0 dl 1786755142 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6010.568404] LustreError: 141540:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6015.575969] LustreError: 141540:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6018.585770] LustreError: 142314:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af8c9621880 x1873542299157888/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:85/0 lens 576/0 e 0 to 0 dl 1786755150 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6018.610206] LustreError: 142314:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 101 previous similar messages [ 6018.622867] LustreError: 142314:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 6028.639219] LustreError: 142314:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6048.331695] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 20:52:48 (1786755168) [ 6146.778987] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 20:54:26 (1786755266) [ 6177.139641] LustreError: 141539:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af9f15bc700 x1873542299215872/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:244/0 lens 576/0 e 0 to 0 dl 1786755309 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6177.176239] LustreError: 141539:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 117 previous similar messages [ 6177.192893] LustreError: 141539:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6177.631155] LustreError: 141539:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6193.687240] LustreError: 141538:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6193.699306] LustreError: 141538:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 36 previous similar messages [ 6209.303634] LustreError: 143782:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9af9fe2adf80 x1873542299233792/t0(0) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:276/0 lens 576/0 e 0 to 0 dl 1786755341 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 6209.333369] LustreError: 143782:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 76 previous similar messages [ 6209.357976] LustreError: 143782:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6209.379590] LustreError: 143782:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 76 previous similar messages [ 6230.729832] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 20:55:50 (1786755350) [ 6268.718535] Lustre: DEBUG MARKER: phase 2 [ 6279.657806] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 20:56:39 (1786755399) [ 6361.019494] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 20:58:01 (1786755481) [ 6362.475991] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6364.029382] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 20:58:04 (1786755484) [ 6368.251345] Lustre: DEBUG MARKER: Started rundbench load pid=128786 ... [ 6372.424392] LustreError: 148360:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 6372.428233] LustreError: 148360:0:(osd_handler.c:715:osd_ro()) Skipped 3 previous similar messages [ 6373.448893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6375.901103] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6377.749677] Lustre: Failing over lustre-MDT0000 [ 6377.855468] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 6377.986371] Lustre: server umount lustre-MDT0000 complete [ 6397.151854] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786755502/real 1786755502] req@ffff9af8c10b0e00 x1873542304841344/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786755518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6397.185846] Lustre: 3293:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 32 previous similar messages [ 6397.202333] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.210624] LustreError: Skipped 5 previous similar messages [ 6397.402701] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.427240] Lustre: Skipped 13 previous similar messages [ 6408.297943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.301807] Lustre: Skipped 8 previous similar messages [ 6408.435171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6408.444441] Lustre: Skipped 8 previous similar messages [ 6413.900902] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6414.557758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 6414.570390] Lustre: Skipped 8 previous similar messages [ 6415.847541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5427 to 0x280000400:5473) [ 6415.852582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5480 to 0x240000400:5505) [ 6422.439310] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6422.446810] Lustre: Skipped 13 previous similar messages [ 6424.371349] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6426.106209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6433.724867] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6436.701770] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 6438.833765] Lustre: Failing over lustre-MDT0000 [ 6438.900502] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 6439.073785] Lustre: server umount lustre-MDT0000 complete [ 6463.127963] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6475.508452] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 6475.512863] Lustre: Skipped 1 previous similar message [ 6475.543806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5534 to 0x240000400:5569) [ 6475.547888] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5502 to 0x280000400:5537) [ 6482.139171] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6483.747964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6518.558316] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 21:00:38 (1786755638) [ 6643.731480] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6655.963987] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6657.901527] Lustre: Failing over lustre-MDT0000 [ 6658.356312] Lustre: server umount lustre-MDT0000 complete [ 6680.788679] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6699.509458] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6342 to 0x280000400:6369) [ 6699.509795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6374 to 0x240000400:6401) [ 6705.940646] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6707.365841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6776.940384] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 21:04:57 (1786755897) [ 6778.027283] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 6779.397140] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:04:59 (1786755899) [ 6780.874790] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 6782.314037] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:05:02 (1786755902) [ 6789.808453] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6792.162083] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6793.737978] Lustre: Failing over lustre-OST0000 [ 6793.806624] Lustre: server umount lustre-OST0000 complete [ 6796.138482] LustreError: 36582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.164398] LustreError: 36582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6796.768990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6806.375432] LustreError: 14851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.388674] LustreError: 14851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6817.729233] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6827.154393] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.247444] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6840.989542] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6843.625771] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6845.380396] Lustre: Failing over lustre-OST0000 [ 6845.444293] Lustre: server umount lustre-OST0000 complete [ 6847.353405] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.380871] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6870.725864] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6881.193338] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.129206] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6894.129583] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:06:54 (1786756014) [ 6895.633874] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 6897.567523] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:06:57 (1786756017) [ 6901.055474] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6904.259234] Lustre: Failing over lustre-MDT0000 [ 6904.484749] Lustre: server umount lustre-MDT0000 complete [ 6924.167240] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6928.102773] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6939.523314] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 6939.775564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6584 to 0x280000400:6625) [ 6939.779671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6617 to 0x240000400:6657) [ 6947.491232] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6949.096983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6957.498824] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:07:57 (1786756077) [ 6961.334514] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6963.865673] Lustre: Failing over lustre-MDT0000 [ 6964.212516] Lustre: server umount lustre-MDT0000 complete [ 6988.208674] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6990.717669] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6990.720631] LustreError: 159243:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9af8d2bea300 x1873542307919104/t335007449091(335007449091) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:302/0 lens 592/608 e 0 to 0 dl 1786756122 ref 1 fl Interpret:/604/0 rc 301/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 7007.100710] Lustre: lustre-MDT0000: Client 2513f734-913c-483a-be19-9bc2215f95a9 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 7007.115094] Lustre: 159243:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9af9fe59e300 x1873542307919104/t335007449091(335007449091) o101->2513f734-913c-483a-be19-9bc2215f95a9@192.168.202.49@tcp:319/0 lens 592/3488 e 0 to 0 dl 1786756139 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 7007.250160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6659 to 0x240000400:6689) [ 7007.253644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6584 to 0x280000400:6657) [ 7013.254501] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7014.710987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7023.289836] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:09:03 (1786756143) [ 7027.883228] Lustre: Failing over lustre-OST0000 [ 7027.966290] Lustre: server umount lustre-OST0000 complete [ 7028.202727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.218591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.232695] Lustre: Skipped 11 previous similar messages [ 7028.234903] LustreError: 36588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.256883] LustreError: 36588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7031.752240] Lustre: Failing over lustre-MDT0000 [ 7032.245883] Lustre: server umount lustre-MDT0000 complete [ 7050.073001] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.078774] LustreError: Skipped 4 previous similar messages [ 7050.413708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7050.419595] Lustre: Skipped 6 previous similar messages [ 7050.601037] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6584 to 0x280000400:6689) [ 7051.359668] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756155/real 1786756155] req@ffff9af8c5cc9500 x1873542305970688/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786756171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7051.410120] Lustre: 3292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 41 previous similar messages [ 7054.031930] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7055.841933] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7055.860773] Lustre: Skipped 11 previous similar messages [ 7061.776135] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7061.788803] Lustre: Skipped 6 previous similar messages [ 7063.573282] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7063.578434] Lustre: Skipped 6 previous similar messages [ 7063.584449] Lustre: lustre-OST0000: Denying connection for new client 9789f57b-e6df-4b7d-90c7-265c60ca08cd (at 192.168.202.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7063.608646] Lustre: Skipped 11 previous similar messages [ 7067.188687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6659 to 0x240000400:6721) [ 7068.309393] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7079.347073] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:09:59 (1786756199) [ 7081.154358] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 7083.106885] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:10:03 (1786756203) [ 7084.617512] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 7086.446966] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:10:06 (1786756206) [ 7087.823892] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 7089.955985] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:10:09 (1786756209) [ 7091.756990] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 7093.324595] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:10:13 (1786756213) [ 7094.663456] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 7096.143305] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:10:16 (1786756216) [ 7097.572725] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 7099.179923] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:10:19 (1786756219) [ 7100.757922] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 7102.581809] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:10:22 (1786756222) [ 7104.155246] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 7105.600257] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:10:26 (1786756226) [ 7107.113316] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 7108.689315] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:10:28 (1786756228) [ 7110.274241] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 7111.920416] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:10:32 (1786756232) [ 7113.445369] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 7115.520640] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 21:10:35 (1786756235) [ 7117.235971] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 7118.990411] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 21:10:39 (1786756239) [ 7120.644357] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 7122.425497] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 21:10:42 (1786756242) [ 7124.166252] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 7125.828348] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 21:10:46 (1786756246) [ 7127.691819] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 7129.726059] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 21:10:49 (1786756249) [ 7131.348459] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 7133.212296] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 21:10:53 (1786756253) [ 7134.949310] Lustre: 163848:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 9789f57b-e6df-4b7d-90c7-265c60ca08cd at adminstrative request [ 7145.634855] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 21:11:06 (1786756266) [ 7153.383239] Lustre: Failing over lustre-MDT0000 [ 7153.522639] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 7153.718756] Lustre: server umount lustre-MDT0000 complete [ 7184.490819] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7194.420741] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 7194.430503] Lustre: Skipped 6 previous similar messages [ 7194.470803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6741 to 0x280000400:6785) [ 7194.472681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6773 to 0x240000400:6817) [ 7199.672746] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7201.339735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7209.747817] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 21:12:09 (1786756329) [ 7223.741175] Lustre: Failing over lustre-OST0000 [ 7223.867882] Lustre: server umount lustre-OST0000 complete [ 7224.167965] LustreError: 36584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.184908] LustreError: 36584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7225.312587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7248.952666] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7259.281974] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7261.219095] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7270.761400] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 21:13:11 (1786756391) [ 7275.483330] Lustre: Failing over lustre-MDT0000 [ 7275.828055] Lustre: server umount lustre-MDT0000 complete [ 7282.991206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6918 to 0x240000400:6945) [ 7282.995880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6741 to 0x280000400:6817) [ 7288.008675] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7295.906708] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 21:13:36 (1786756416) [ 7299.828849] LustreError: 168295:0:(osd_handler.c:715:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 7299.835578] LustreError: 168295:0:(osd_handler.c:715:osd_ro()) Skipped 6 previous similar messages [ 7300.862320] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7303.736100] Lustre: Failing over lustre-OST0000 [ 7303.792901] Lustre: server umount lustre-OST0000 complete [ 7308.255819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7320.934049] Lustre: lustre-OST0000: Not available for connect from 192.168.202.49@tcp (not set up) [ 7328.897188] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7337.719811] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.481553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7347.824625] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 21:14:28 (1786756468) [ 7351.837588] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7355.640830] Lustre: Failing over lustre-OST0000 [ 7355.760076] Lustre: server umount lustre-OST0000 complete [ 7356.795711] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7356.810093] LustreError: 9174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 7375.974072] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.49@tcp inode [0x2000284a1:0x5:0x0] object 0x240000400:6947 extent [0-1048575]: client csum 8b5ba24d, server csum e5db169e [ 7379.456568] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7388.220678] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7389.988560] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7399.665604] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 21:15:19 (1786756519) [ 7403.943925] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7407.813298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7413.879354] Lustre: Failing over lustre-MDT0000 [ 7414.232849] Lustre: server umount lustre-MDT0000 complete [ 7418.203432] LustreError: 36592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756539 with bad export cookie 5292905616947650538 [ 7428.578764] Lustre: Failing over lustre-OST0000 [ 7428.631456] Lustre: server umount lustre-OST0000 complete [ 7463.713372] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af9c5e94000 x1873542306089856/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7469.705676] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7479.438775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6856 to 0x280000400:6881) [ 7494.271598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6948 to 0x240000400:6977) [ 7495.093559] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7510.707075] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 21:17:10 (1786756630) [ 7528.213969] Lustre: Failing over lustre-OST0000 [ 7529.830200] Lustre: lustre-OST0000: Not available for connect from 192.168.202.49@tcp (stopping) [ 7530.343280] Lustre: server umount lustre-OST0000 complete [ 7535.165431] Lustre: Failing over lustre-MDT0000 [ 7535.627789] Lustre: server umount lustre-MDT0000 complete [ 7555.716173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6856 to 0x280000400:6913) [ 7557.815605] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7571.269982] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7573.423056] Lustre: lustre-OST0000: Denying connection for new client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 7573.440875] Lustre: Skipped 1 previous similar message [ 7593.822573] Lustre: lustre-OST0000: Denying connection for new client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 7593.849106] Lustre: Skipped 3 previous similar messages [ 7629.664442] Lustre: lustre-OST0000: Denying connection for new client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:10 [ 7629.672560] Lustre: Skipped 6 previous similar messages [ 7640.500335] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7640.510963] Lustre: 176185:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 45a1e0c9-da0c-4630-b698-438eb9e78a10@ [ 7640.535775] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7640.669843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6988 to 0x240000400:7009) [ 7648.698952] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 65 sec [ 7663.753804] Lustre: DEBUG MARKER: free_before: 7520256 free_after: 7520256 [ 7670.578737] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 21:19:50 (1786756790) [ 7675.198905] Lustre: Failing over lustre-OST0000 [ 7675.329488] Lustre: server umount lustre-OST0000 complete [ 7675.776523] LustreError: 6692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.800570] LustreError: 6692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 33 previous similar messages [ 7676.399741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7676.405207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.417142] Lustre: Skipped 11 previous similar messages [ 7694.613508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.617776] Lustre: Skipped 10 previous similar messages [ 7694.635322] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7694.643304] Lustre: Skipped 8 previous similar messages [ 7696.234718] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7696.243898] Lustre: Skipped 8 previous similar messages [ 7696.550889] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7696.562133] Lustre: Skipped 12 previous similar messages [ 7701.329437] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7714.761045] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 21:20:33 (1786756833) [ 7719.916039] Lustre: Failing over lustre-OST0000 [ 7719.982973] Lustre: server umount lustre-OST0000 complete [ 7741.808199] LustreError: 179269:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7741.842357] LustreError: 179269:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 42 previous similar messages [ 7746.130944] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7747.551501] Lustre: *** cfs_fail_loc=715, val=40*** [ 7748.575358] Lustre: *** cfs_fail_loc=715, val=40*** [ 7756.768162] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756862/real 1786756862] req@ffff9af8cd02aa00 x1873542306165760/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786756878 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7756.823230] Lustre: 3291:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [ 7756.836553] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 1:25 [ 7758.176381] Lustre: lustre-OST0000: Client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [ 7762.913103] Lustre: *** cfs_fail_loc=715, val=40*** [ 7773.153657] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 2 clients in recovery for 1:09 [ 7779.296316] Lustre: *** cfs_fail_loc=715, val=40*** [ 7779.304563] Lustre: Skipped 1 previous similar message [ 7781.911339] LustreError: 179269:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7781.916340] LustreError: 179269:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 82 previous similar messages [ 7790.591349] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7792.474755] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7802.039111] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 21:22:01 (1786756921) [ 7807.283042] Lustre: Failing over lustre-MDT0000 [ 7807.570633] Lustre: server umount lustre-MDT0000 complete [ 7824.735581] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7824.741789] LustreError: Skipped 4 previous similar messages [ 7834.911961] LustreError: 3291:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9af8ce737b80 x1873542306181120/t0(0) o250->MGC192.168.202.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7839.447124] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7849.873564] LustreError: 180871:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7856.095888] Lustre: *** cfs_fail_loc=715, val=80*** [ 7856.101668] Lustre: Skipped 1 previous similar message [ 7866.206516] Lustre: lustre-MDT0000: Client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 7866.215772] Lustre: Skipped 1 previous similar message [ 7872.479549] Lustre: *** cfs_fail_loc=715, val=80*** [ 7881.567031] Lustre: lustre-MDT0000: Client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:37 [ 7897.949064] Lustre: lustre-MDT0000: Client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:21 [ 7904.223401] Lustre: *** cfs_fail_loc=715, val=80*** [ 7904.228843] Lustre: Skipped 1 previous similar message [ 7914.333722] Lustre: lustre-MDT0000: Client bd7e002d-0978-4214-9df8-a43c597f6a65 (at 192.168.202.49@tcp) reconnected, waiting for 1 clients in recovery for 0:05 [ 7929.707555] Lustre: lustre-MDT0000: Recovery already passed deadline 0:10. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 7929.967678] LustreError: 180871:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7930.187730] Lustre: 180871:0:(ldlm_lib.c:2951:target_recovery_thread()) too long recovery - read logs [ 7930.203468] LustreError: dumping log to /tmp/lustre-log.1786757051.180871 [ 7930.370538] Lustre: lustre-MDT0000: Recovery over after 1:21, of 1 clients 1 recovered and 0 were evicted. [ 7930.379475] Lustre: Skipped 9 previous similar messages [ 7930.419354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6926 to 0x280000400:6945) [ 7930.422556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:7023 to 0x240000400:7041) [ 7936.914453] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7938.893260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7948.363981] Lustre: DEBUG MARKER: == replay-single test complete, duration 7739 sec ======== 21:24:28 (1786757068) [ 7950.117406] Lustre: DEBUG MARKER: === replay-single: start cleanup 21:24:30 (1786757070) === [ 7960.721252] Lustre: DEBUG MARKER: === replay-single: finish cleanup 21:24:40 (1786757080) === [ 7993.396588] Lustre: server umount lustre-MDT0000 complete [ 7998.411023] LustreError: 36751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757119 with bad export cookie 5292905616947662956 [ 8008.791659] Lustre: server umount lustre-OST0000 complete [ 8013.640898] Lustre: server umount lustre-OST0001 complete [ 8028.593156] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8032.164076] Key type lgssc unregistered [ 8032.583517] LNet: 183007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8032.590826] LNetError: 183007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8032.614742] LNet: Removed LNI 192.168.202.149@tcp [ 8033.667181] Key type .llcrypt unregistered [ 8033.670228] Key type ._llcrypt unregistered