[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493798454 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002217] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006311] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011018] SELinux: Initializing. [ 0.011739] *** VALIDATE selinux *** [ 0.018106] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022780] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024014] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026091] *** VALIDATE tmpfs *** [ 0.028169] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.032213] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033127] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035023] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039727] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.041226] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042645] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049007] ... event mask: 000000070000000f [ 0.050247] rcu: Hierarchical SRCU implementation. [ 0.052689] smp: Bringing up secondary CPUs ... [ 0.053645] x86: Booting SMP configuration: [ 0.054016] .... node #0, CPUs: #1 #2 #3 [ 0.058607] smp: Brought up 1 node, 4 CPUs [ 0.060008] smpboot: Max logical packages: 1 [ 0.060994] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.174028] node 0 deferred pages initialised in 112ms [ 0.179011] devtmpfs: initialized [ 0.180491] x86/mm: Memory block size: 128MB [ 0.185596] gcov: version magic: 0x41383552 [ 0.188338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.189228] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.190479] pinctrl core: initialized pinctrl subsystem [ 0.191413] [ 0.192010] ************************************************************* [ 0.193023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194025] ** ** [ 0.195022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.196018] ** ** [ 0.197022] ** This means that this kernel is built to expose internal ** [ 0.198022] ** IOMMU data structures, which may compromise security on ** [ 0.199022] ** your system. ** [ 0.200025] ** ** [ 0.201024] ** If you see this message and you are not debugging the ** [ 0.202021] ** kernel, report this immediately to your vendor! ** [ 0.203031] ** ** [ 0.204020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205024] ************************************************************* [ 0.207261] NET: Registered protocol family 16 [ 0.208485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211653] cpuidle: using governor menu [ 0.213750] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215533] PCI: Using configuration type 1 for base access [ 0.219281] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239203] cryptd: max_cpu_qlen set to 1000 [ 0.243314] ACPI: Added _OSI(Module Device) [ 0.244011] ACPI: Added _OSI(Processor Device) [ 0.246069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.247109] ACPI: Added _OSI(Processor Aggregator Device) [ 0.254514] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261673] ACPI: Interpreter enabled [ 0.262047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.263009] ACPI: Using IOAPIC for interrupt routing [ 0.264094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265441] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275131] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279905] acpiphp: Slot [2] registered [ 0.280110] acpiphp: Slot [5] registered [ 0.281086] acpiphp: Slot [6] registered [ 0.282058] acpiphp: Slot [7] registered [ 0.283033] acpiphp: Slot [8] registered [ 0.283989] acpiphp: Slot [9] registered [ 0.284097] acpiphp: Slot [10] registered [ 0.285154] acpiphp: Slot [3] registered [ 0.286066] acpiphp: Slot [4] registered [ 0.287135] acpiphp: Slot [11] registered [ 0.288000] acpiphp: Slot [12] registered [ 0.290087] acpiphp: Slot [13] registered [ 0.292107] acpiphp: Slot [14] registered [ 0.294098] acpiphp: Slot [15] registered [ 0.297115] acpiphp: Slot [16] registered [ 0.300157] acpiphp: Slot [17] registered [ 0.302238] acpiphp: Slot [18] registered [ 0.307217] acpiphp: Slot [19] registered [ 0.310113] acpiphp: Slot [20] registered [ 0.312273] acpiphp: Slot [21] registered [ 0.314141] acpiphp: Slot [22] registered [ 0.317196] acpiphp: Slot [23] registered [ 0.319158] acpiphp: Slot [24] registered [ 0.321163] acpiphp: Slot [25] registered [ 0.323542] acpiphp: Slot [26] registered [ 0.326319] acpiphp: Slot [27] registered [ 0.328275] acpiphp: Slot [28] registered [ 0.330097] acpiphp: Slot [29] registered [ 0.332084] acpiphp: Slot [30] registered [ 0.333167] acpiphp: Slot [31] registered [ 0.335068] PCI host bridge to bus 0000:00 [ 0.335914] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352876] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356348] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385962] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390585] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406851] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416058] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.428016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452913] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.492489] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545903] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.556114] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.566016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.602685] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618207] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.643016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.652402] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658010] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.663014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.679020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690260] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691227] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694149] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709121] iommu: Default domain type: Passthrough [ 0.710497] SCSI subsystem initialized [ 0.712203] ACPI: bus type USB registered [ 0.713078] usbcore: registered new interface driver usbfs [ 0.715037] usbcore: registered new interface driver hub [ 0.716055] usbcore: registered new device driver usb [ 0.717119] pps_core: LinuxPPS API ver. 1 registered [ 0.719007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721030] PTP clock support registered [ 0.722172] EDAC MC: Ver: 3.0.0 [ 0.724403] PCI: Using ACPI for IRQ routing [ 0.726607] NetLabel: Initializing [ 0.727008] NetLabel: domain hash size = 128 [ 0.728005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729039] NetLabel: unlabeled traffic allowed by default [ 0.731212] vgaarb: loaded [ 0.732426] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.734009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.742511] clocksource: Switched to clocksource kvm-clock [ 0.885146] VFS: Disk quotas dquot_6.6.0 [ 0.887013] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889791] *** VALIDATE ramfs *** [ 0.891071] *** VALIDATE hugetlbfs *** [ 0.892599] pnp: PnP ACPI init [ 0.894963] pnp: PnP ACPI: found 6 devices [ 0.910230] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.913920] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915659] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917682] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.920306] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.922397] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.925612] NET: Registered protocol family 2 [ 0.928150] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.933623] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.936983] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.942515] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946196] TCP: Hash tables configured (established 65536 bind 65536) [ 0.949210] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.952761] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956152] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959647] NET: Registered protocol family 1 [ 0.962584] RPC: Registered named UNIX socket transport module. [ 0.964856] RPC: Registered udp transport module. [ 0.966770] RPC: Registered tcp transport module. [ 0.968598] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971245] NET: Registered protocol family 44 [ 0.972977] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.975666] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.977429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.980110] PCI: CLS 0 bytes, default 64 [ 0.982120] Unpacking initramfs... [ 2.852618] debug: unmapping init [mem 0xffff9c5cbcc54000-0xffff9c5cbffbffff] [ 2.857192] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.859860] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.863101] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.517468] Initialise system trusted keyrings [ 3.520507] Key type blacklist registered [ 3.522421] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.535177] zbud: loaded [ 3.539255] *** VALIDATE nfs *** [ 3.540852] *** VALIDATE nfs4 *** [ 3.544361] pstore: using deflate compression [ 3.548874] Platform Keyring initialized [ 3.706769] NET: Registered protocol family 38 [ 3.709805] Key type asymmetric registered [ 3.712549] Asymmetric key parser 'x509' registered [ 3.715557] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.721181] io scheduler mq-deadline registered [ 3.722807] io scheduler kyber registered [ 3.725525] io scheduler bfq registered [ 3.729308] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.732989] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.738458] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.742310] ACPI: Power Button [PWRF] [ 3.749596] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.758634] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.784438] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.803157] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.833224] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.865339] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.903707] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.908238] Non-volatile memory driver v1.3 [ 3.910237] Linux agpgart interface v0.103 [ 3.949894] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.952855] vda: detected capacity change from 0 to 74694656 [ 3.970315] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.975167] vdb: detected capacity change from 0 to 1073741824 [ 3.996560] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.000766] vdc: detected capacity change from 0 to 2621440000 [ 4.025270] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.030091] vdd: detected capacity change from 0 to 2621440000 [ 4.056119] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.062870] vde: detected capacity change from 0 to 4294967296 [ 4.097558] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.101776] vdf: detected capacity change from 0 to 4294967296 [ 4.119525] libphy: Fixed MDIO Bus: probed [ 4.129171] usbcore: registered new interface driver usbserial_generic [ 4.132143] usbserial: USB Serial support registered for generic [ 4.135741] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.141764] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.144473] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.148705] mousedev: PS/2 mouse device common for all mice [ 4.151793] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.155960] rtc_cmos 00:05: RTC can wake from S4 [ 4.161566] rtc_cmos 00:05: registered as rtc0 [ 4.163448] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.166355] intel_pstate: CPU model not supported [ 4.172023] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.176949] hid: raw HID events driver (C) Jiri Kosina [ 4.181125] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.182421] usbcore: registered new interface driver usbhid [ 4.182424] usbhid: USB HID core driver [ 4.182531] drop_monitor: Initializing network drop monitor service [ 4.182719] Initializing XFRM netlink socket [ 4.183764] NET: Registered protocol family 10 [ 4.203173] Segment Routing with IPv6 [ 4.205213] NET: Registered protocol family 17 [ 4.207725] mpls_gso: MPLS GSO support [ 4.213977] RAS: Correctable Errors collector initialized. [ 4.217145] AVX version of gcm_enc/dec engaged. [ 4.220382] AES CTR mode by8 optimization enabled [ 4.333175] sched_clock: Marking stable (4333014988, 0)->(5326669304, -993654316) [ 4.340282] registered taskstats version 1 [ 4.343740] Loading compiled-in X.509 certificates [ 4.348488] zswap: loaded using pool lzo/zbud [ 4.386284] Key type big_key registered [ 4.403787] Key type encrypted registered [ 4.405873] ima: No TPM chip found, activating TPM-bypass! [ 4.409527] ima: Allocated hash algorithm: sha1 [ 4.411746] ima: No architecture policies found [ 4.414628] evm: Initialising EVM extended attributes: [ 4.417168] evm: security.selinux [ 4.419215] evm: security.ima [ 4.420603] evm: security.capability [ 4.422757] evm: HMAC attrs: 0x1 [ 4.426581] rtc_cmos 00:05: setting system clock to 2026-08-11 16:01:27 UTC (1786464087) [ 4.435316] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 4.439162] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 4.447090] Write protecting the kernel read-only data: 28672k [ 4.451795] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 4.454880] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 4.500755] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.513489] systemd[1]: Detected virtualization kvm. [ 4.516565] systemd[1]: Detected architecture x86-64. [ 4.519588] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.548611] systemd[1]: No hostname configured. [ 4.550873] systemd[1]: Set hostname to . [ 4.554597] random: systemd: uninitialized urandom read (16 bytes read) [ 4.557548] systemd[1]: Initializing machine ID from random generator. [ 4.833332] random: systemd: uninitialized urandom read (16 bytes read) [ 4.835903] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.842028] random: systemd: uninitialized urandom read (16 bytes read) [ 4.845104] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.857232] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.030857] device-mapper: uevent: version 1.0.3 [ 6.034689] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.418007] random: fast init done Starting dracut initqueue hook... [ 7.556736] virtio_net virtio0 ens2: renamed from eth0 [ 7.748251] scsi host0: ata_piix [ 7.814387] scsi host1: ata_piix [ 7.815717] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.817948] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.977723] random: crng init done [ 12.985819] random: 7 urandom warning(s) missed due to ratelimiting [ 13.265443] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.758165] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.557110] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.112296] SELinux: Disabled at runtime. [ 17.204895] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.213212] systemd[1]: Detected virtualization kvm. [ 17.215118] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.052282] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.054937] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.061357] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.064280] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.068184] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.077773] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.085588] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 18.384861] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.241200] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.236973] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.245647] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.807419] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.023761] EDAC sbridge: Ver: 1.1.2 [ 24.541180] Key type dns_resolver registered [* ] A start [ 24.917373] NFS: Registering the id_resolver key type job is running for Configur…-only root support[ 24.924264] Key type id_resolver registered (6s / no limit)[ 24.930486] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 48.232264] spl: loading out-of-tree module taints kernel. [ 50.663144] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.603587] Key type ._llcrypt registered [ 55.604930] Key type .llcrypt registered [ 55.652154] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 62.969544] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 63.478623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.484756] alg: No test for adler32 (adler32-zlib) [ 64.459983] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 64.790426] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 66.417221] Key type lgssc registered [ 67.017563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.655585] vdc: vdc1 vdc9 [ 74.545094] vde: vde1 vde9 [ 78.186264] vdf: vdf1 vdf9 [ 85.309565] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 89.102805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 90.205952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 90.309759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 90.347681] Lustre: lustre-MDT0000: new disk, initializing [ 90.436848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.453783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 91.892209] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.349592] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 96.581505] Lustre: lustre-OST0000: new disk, initializing [ 96.583644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 96.586556] Lustre: Skipped 1 previous similar message [ 96.612821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 98.677500] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.309299] Lustre: lustre-OST0001: new disk, initializing [ 103.312662] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 103.376622] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.389192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 103.395435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 103.446804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 105.778143] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.545429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 108.548613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 108.584045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 111.917200] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.703707] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 120.944851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.440222] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 123.912917] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 124.788047] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 125.651853] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 126.216576] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Aug 11 12:03:29 EDT 2026 [ 131.823945] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 132.501062] Lustre: DEBUG MARKER: === replay-single: start setup 12:03:35 (1786464215) === [ 134.455377] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing check_config_client /mnt/lustre [ 140.695371] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 142.006809] Lustre: 11289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 143.414374] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.737715] Lustre: DEBUG MARKER: === replay-single: finish setup 12:03:47 (1786464227) === [ 145.340994] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 12:03:48 (1786464228) [ 146.437481] LustreError: 11783:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 146.768979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 147.467474] Lustre: Failing over lustre-MDT0000 [ 147.565616] Lustre: server umount lustre-MDT0000 complete [ 160.670847] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 160.832844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 160.839063] Lustre: Skipped 1 previous similar message [ 160.882263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.309449] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.345481] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 162.368755] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 164.960096] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464232/real 1786464232] req@ffff9c5d034c5880 x1873243429441664/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464248 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 164.970736] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 166.370950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 166.923241] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 167.675392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 169.952126] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464237/real 1786464237] req@ffff9c5d3f96d180 x1873243429442048/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 171.444689] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 12:04:14 (1786464254) [ 172.275186] Lustre: Failing over lustre-OST0000 [ 172.316802] Lustre: server umount lustre-OST0000 complete [ 175.072172] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464242/real 1786464242] req@ffff9c5d3df07b80 x1873243429442560/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 175.084162] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 176.096950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 176.100145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 181.349781] LustreError: 8203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.355916] LustreError: 8203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 185.415742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.468888] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 186.807273] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 186.807392] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 186.813442] Lustre: Skipped 1 previous similar message [ 187.589963] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.880158] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 192.570842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 196.839512] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 12:04:39 (1786464279) [ 198.129268] LustreError: 14793:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 198.571298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 199.513129] Lustre: Failing over lustre-MDT0000 [ 199.665848] Lustre: server umount lustre-MDT0000 complete [ 212.962393] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 213.113052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.175289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.041556] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.094167] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 216.100623] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 216.224148] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464283/real 1786464283] req@ffff9c5d3fedd180 x1873243429464704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 218.594808] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 221.285369] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 221.344245] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464288/real 1786464288] req@ffff9c5d18451180 x1873243429465088/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 221.354545] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 226.405117] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 231.527274] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 236.645111] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 246.886514] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:29 [ 246.897719] Lustre: Skipped 1 previous similar message [ 267.365548] Lustre: lustre-MDT0000: Denying connection for new client d1cb8b92-3187-43d5-9a59-d6a61bd759fc (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:09 [ 267.372037] Lustre: Skipped 3 previous similar messages [ 276.500238] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 276.502908] Lustre: 15424:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 93cc759b-08ff-4bfa-8fd4-e7e68a5dcfa4@ [ 276.509880] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 276.529540] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 276.549585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:65) [ 276.549866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:65) [ 282.304896] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 12:06:04 (1786464364) [ 283.591503] LustreError: 16159:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 283.960044] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 284.898813] Lustre: Failing over lustre-MDT0000 [ 285.037110] Lustre: server umount lustre-MDT0000 complete [ 297.995503] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.129838] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.136270] Lustre: Skipped 2 previous similar messages [ 298.168821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 299.921113] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 300.703179] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 300.706557] Lustre: lustre-MDT0000: Denying connection for new client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 300.712357] Lustre: Skipped 1 previous similar message [ 301.217900] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464368/real 1786464368] req@ffff9c5d1e85ad80 x1873243429487744/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464384 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 301.227908] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 303.586356] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 303.589263] Lustre: Skipped 1 previous similar message [ 360.500486] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 360.506536] Lustre: 16791:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d1cb8b92-3187-43d5-9a59-d6a61bd759fc@ [ 360.519089] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 360.591514] Lustre: lustre-MDT0000: Recovery over after 1:00, of 1 clients 0 recovered and 1 was evicted. [ 360.610504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:97) [ 360.610946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:97) [ 366.811134] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 12:07:29 (1786464449) [ 367.885980] LustreError: 17531:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 368.212441] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 368.907978] Lustre: Failing over lustre-MDT0000 [ 369.016443] Lustre: server umount lustre-MDT0000 complete [ 381.859367] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.980363] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.989633] Lustre: Skipped 1 previous similar message [ 382.024219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.575789] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 382.604563] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 382.622498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:44 to 0x240000400:129) [ 382.627217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:129) [ 383.401399] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 386.080124] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464453/real 1786464453] req@ffff9c5d1e5ff100 x1873243429508096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 386.092751] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 387.043643] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 387.046343] Lustre: Skipped 1 previous similar message [ 387.394954] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 388.030142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 391.808383] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 12:07:54 (1786464474) [ 393.060892] LustreError: 19101:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 393.486544] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 394.262239] Lustre: Failing over lustre-MDT0000 [ 394.451646] Lustre: server umount lustre-MDT0000 complete [ 407.480037] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.520418] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.520689] LustreError: 19705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.524347] Lustre: Skipped 1 previous similar message [ 407.532771] LustreError: 19705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 408.181288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 408.232736] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 408.251367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 408.251368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:161) [ 408.999645] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 412.641831] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 412.643620] Lustre: Skipped 1 previous similar message [ 412.734423] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 413.238520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 416.489713] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 12:08:19 (1786464499) [ 417.442779] LustreError: 20676:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 417.743831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 418.402068] Lustre: Failing over lustre-MDT0000 [ 418.410905] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 418.625683] Lustre: server umount lustre-MDT0000 complete [ 419.424180] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464485/real 1786464485] req@ffff9c5d3e208000 x1873243429521152/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 419.433023] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 431.581194] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 431.715476] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.724522] Lustre: Skipped 1 previous similar message [ 433.353830] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 437.217939] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 437.221466] Lustre: Skipped 1 previous similar message [ 437.860960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 437.898497] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 437.916657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:193) [ 437.916667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 439.877421] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.531739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 444.379239] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 12:08:47 (1786464527) [ 445.409370] LustreError: 22250:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 445.731550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 446.389432] Lustre: Failing over lustre-MDT0000 [ 446.601943] Lustre: server umount lustre-MDT0000 complete [ 459.375928] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 459.520323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.522379] Lustre: Skipped 2 previous similar messages [ 460.884284] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 462.974430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:225) [ 462.974733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 464.875301] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.422434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 469.282806] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 12:09:12 (1786464552) [ 470.704308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 471.490134] Lustre: Failing over lustre-MDT0000 [ 471.672123] Lustre: server umount lustre-MDT0000 complete [ 484.816470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.822092] Lustre: Skipped 2 previous similar messages [ 484.979205] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 484.982247] Lustre: Skipped 1 previous similar message [ 485.017447] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 485.020763] Lustre: Skipped 1 previous similar message [ 485.036292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:257) [ 485.036323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:257) [ 486.280035] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 489.955377] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 489.957774] Lustre: Skipped 3 previous similar messages [ 490.597657] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.976099] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464558/real 1786464558] req@ffff9c5d3df05f80 x1873243429556352/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 490.995753] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 491.284382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 495.121812] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 12:09:37 (1786464577) [ 495.486329] Lustre: *** cfs_fail_loc=13b, val=315*** [ 495.488502] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 495.490093] LustreError: 24425:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d3e1e6a00 x1873243422164224/t38654705666(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:205/0 lens 392/456 e 0 to 0 dl 1786464595 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 497.477703] LustreError: 25445:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 497.480559] LustreError: 25445:0:(osd_handler.c:720:osd_ro()) Skipped 1 previous similar message [ 497.805188] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 498.522421] Lustre: Failing over lustre-MDT0000 [ 498.687603] Lustre: server umount lustre-MDT0000 complete [ 511.496855] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.500444] LustreError: Skipped 1 previous similar message [ 513.303695] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 515.419879] Lustre: 26051:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d18453480 x1873243422164224/t38654705666(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:225/0 lens 392/456 e 0 to 0 dl 1786464615 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 515.425230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:289) [ 515.427224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:289) [ 517.522267] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.155277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 522.001343] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 12:10:04 (1786464604) [ 523.765428] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 524.577107] Lustre: Failing over lustre-MDT0000 [ 524.727399] Lustre: server umount lustre-MDT0000 complete [ 539.675952] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 541.337218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:321) [ 541.337218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:321) [ 543.844963] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.446425] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 548.287240] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 12:10:31 (1786464631) [ 549.600269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 549.934869] Lustre: *** cfs_fail_loc=114, val=0*** [ 550.911749] Lustre: Failing over lustre-MDT0000 [ 551.123158] Lustre: server umount lustre-MDT0000 complete [ 564.372155] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.378480] Lustre: Skipped 6 previous similar messages [ 566.607801] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 566.910115] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 566.923747] Lustre: Skipped 2 previous similar messages [ 567.005704] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 567.012457] Lustre: Skipped 2 previous similar messages [ 567.048541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:353) [ 567.052156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:353) [ 569.832217] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 569.840258] Lustre: Skipped 5 previous similar messages [ 574.295620] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.897022] hrtimer: interrupt took 1658873 ns [ 575.461429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 583.008686] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 12:11:05 (1786464665) [ 586.420123] LustreError: 30280:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 586.433226] LustreError: 30280:0:(osd_handler.c:720:osd_ro()) Skipped 2 previous similar messages [ 587.480256] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 588.475219] Lustre: *** cfs_fail_loc=128, val=0*** [ 591.034162] Lustre: Failing over lustre-MDT0000 [ 591.279080] Lustre: server umount lustre-MDT0000 complete [ 608.152984] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.159911] LustreError: Skipped 2 previous similar messages [ 608.647774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.651739] Lustre: Skipped 4 previous similar messages [ 608.773711] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 612.614722] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 618.183847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:195 to 0x240000400:385) [ 618.183847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:385) [ 620.960199] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464688/real 1786464688] req@ffff9c5d1e5ffb80 x1873243429606656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 621.011867] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 624.295968] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 625.978380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 635.332251] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 12:11:57 (1786464717) [ 638.610536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 640.464242] Lustre: Failing over lustre-MDT0000 [ 640.668542] Lustre: server umount lustre-MDT0000 complete [ 656.159460] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.537731] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 659.287761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:391 to 0x280000400:417) [ 659.288139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:391 to 0x240000400:417) [ 663.798977] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 664.733856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 670.799648] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 12:12:33 (1786464753) [ 673.193145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 674.253140] Lustre: Failing over lustre-MDT0000 [ 674.410791] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 674.500942] Lustre: server umount lustre-MDT0000 complete [ 689.899455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 692.363254] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 693.998812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:423 to 0x280000400:449) [ 693.999260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:423 to 0x240000400:449) [ 698.376931] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 699.348642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 704.548493] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 12:13:07 (1786464787) [ 706.959722] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 713.213416] Lustre: Failing over lustre-MDT0000 [ 713.408732] Lustre: server umount lustre-MDT0000 complete [ 727.997525] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.004042] Lustre: Skipped 7 previous similar messages [ 728.114056] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.706716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 729.711153] Lustre: Skipped 3 previous similar messages [ 730.282566] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 731.990224] Lustre: lustre-MDT0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 731.993633] Lustre: Skipped 3 previous similar messages [ 732.012337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:577) [ 732.014580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:577) [ 733.160501] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 733.165315] Lustre: Skipped 7 previous similar messages [ 736.109663] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 736.937630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 753.772809] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 12:13:56 (1786464836) [ 755.853358] LustreError: 36772:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 755.858087] LustreError: 36772:0:(osd_handler.c:720:osd_ro()) Skipped 3 previous similar messages [ 756.493835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 757.877057] Lustre: Failing over lustre-MDT0000 [ 758.074911] Lustre: server umount lustre-MDT0000 complete [ 773.057168] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 773.062687] LustreError: Skipped 3 previous similar messages [ 773.435558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 775.913777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:609) [ 775.914187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:609) [ 776.265665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 782.229769] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 783.188844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 790.344785] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 12:14:32 (1786464872) [ 792.709471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 794.051691] Lustre: Failing over lustre-MDT0000 [ 794.291149] Lustre: server umount lustre-MDT0000 complete [ 809.446326] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 811.549077] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 811.742249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:641) [ 811.742746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:641) [ 816.899130] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 817.654606] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 822.688338] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 12:15:05 (1786464905) [ 825.071856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 826.126552] Lustre: Failing over lustre-MDT0000 [ 826.396400] Lustre: server umount lustre-MDT0000 complete [ 840.051887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 841.706534] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 842.401603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:673) [ 842.401761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:673) [ 845.921277] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.467339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 850.032503] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 12:15:32 (1786464932) [ 851.514542] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 852.251139] Lustre: Failing over lustre-MDT0000 [ 852.515439] Lustre: server umount lustre-MDT0000 complete [ 866.148953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.151476] Lustre: Skipped 6 previous similar messages [ 867.857869] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 868.030522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:705) [ 868.036194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:705) [ 872.501854] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 873.217982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 877.444233] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 12:16:00 (1786464960) [ 879.145909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 880.000446] Lustre: Failing over lustre-MDT0000 [ 880.160510] Lustre: server umount lustre-MDT0000 complete [ 881.056099] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786464948/real 1786464948] req@ffff9c5d366d4700 x1873243429776512/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786464964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 881.072055] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 893.772335] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 893.775129] Lustre: Skipped 1 previous similar message [ 894.968791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:737) [ 894.969995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:737) [ 895.673303] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 900.493130] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 901.147295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 905.269270] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 12:16:28 (1786464988) [ 906.758889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 907.620522] Lustre: Failing over lustre-MDT0000 [ 907.902147] Lustre: server umount lustre-MDT0000 complete [ 923.149275] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 924.309841] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:769) [ 924.309848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:560 to 0x240000400:769) [ 927.210784] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 927.784950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 931.538567] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 12:16:54 (1786465014) [ 932.930039] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 933.640373] Lustre: Failing over lustre-MDT0000 [ 933.870859] Lustre: server umount lustre-MDT0000 complete [ 948.165845] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 949.937435] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:801) [ 949.937445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:801) [ 952.131164] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.671650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 956.453240] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 12:17:19 (1786465039) [ 957.794265] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 958.471500] Lustre: Failing over lustre-MDT0000 [ 958.633619] Lustre: server umount lustre-MDT0000 complete [ 971.735791] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 971.738080] Lustre: Skipped 2 previous similar messages [ 973.104460] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 975.167204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:771 to 0x240000400:833) [ 975.167703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:833) [ 977.363231] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.918718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 981.580698] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 12:17:44 (1786465064) [ 983.115172] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 983.887570] Lustre: Failing over lustre-MDT0000 [ 984.149685] Lustre: server umount lustre-MDT0000 complete [ 999.513219] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1001.066974] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1001.070064] Lustre: Skipped 8 previous similar messages [ 1001.141595] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1001.144317] Lustre: Skipped 8 previous similar messages [ 1001.159488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:835 to 0x240000400:865) [ 1001.159593] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:865) [ 1002.976715] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.978219] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1002.981747] Lustre: Skipped 17 previous similar messages [ 1002.984179] Lustre: Skipped 17 previous similar messages [ 1003.699926] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.264256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1008.008409] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 12:18:10 (1786465090) [ 1009.472809] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1010.190789] Lustre: Failing over lustre-MDT0000 [ 1010.371817] Lustre: server umount lustre-MDT0000 complete [ 1025.359349] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1026.735349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:835 to 0x240000400:897) [ 1026.735410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:867 to 0x280000400:897) [ 1029.252485] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1029.793077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1033.370219] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 12:18:36 (1786465116) [ 1034.359374] LustreError: 52556:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1034.361603] LustreError: 52556:0:(osd_handler.c:720:osd_ro()) Skipped 9 previous similar messages [ 1034.644523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1035.333847] Lustre: Failing over lustre-MDT0000 [ 1035.596676] Lustre: server umount lustre-MDT0000 complete [ 1048.381559] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.385058] LustreError: Skipped 9 previous similar messages [ 1050.010244] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1052.139520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:899 to 0x240000400:929) [ 1052.144819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:929) [ 1054.266649] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1054.877781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1058.589817] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 12:19:01 (1786465141) [ 1060.075265] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1060.829279] Lustre: Failing over lustre-MDT0000 [ 1061.082483] Lustre: server umount lustre-MDT0000 complete [ 1076.468321] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1077.984926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:931 to 0x280000400:961) [ 1077.985028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:899 to 0x240000400:961) [ 1080.673946] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.315188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1085.199742] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 12:19:28 (1786465168) [ 1086.649078] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1087.416232] Lustre: Failing over lustre-MDT0000 [ 1087.609389] Lustre: server umount lustre-MDT0000 complete [ 1100.873470] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1100.876679] Lustre: Skipped 4 previous similar messages [ 1102.399581] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1103.563555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:963 to 0x240000400:993) [ 1103.563795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:931 to 0x280000400:993) [ 1106.884056] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.528635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1111.334784] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 12:19:54 (1786465194) [ 1112.762344] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1113.587658] Lustre: Failing over lustre-MDT0000 [ 1113.703359] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1113.869403] Lustre: server umount lustre-MDT0000 complete [ 1128.635210] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1134.248787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:995 to 0x240000400:1025) [ 1134.248787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:995 to 0x280000400:1025) [ 1136.228537] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1136.822816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1140.390127] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 12:20:23 (1786465223) [ 1141.743806] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1142.470858] Lustre: Failing over lustre-MDT0000 [ 1142.694131] Lustre: server umount lustre-MDT0000 complete [ 1157.733619] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1159.871134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1057) [ 1159.871134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1027 to 0x240000400:1057) [ 1161.888181] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.434100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1166.112952] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:20:48 (1786465248) [ 1167.471415] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1168.217110] Lustre: Failing over lustre-MDT0000 [ 1168.464626] Lustre: server umount lustre-MDT0000 complete [ 1182.926120] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1185.004302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1089) [ 1185.004318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1059 to 0x240000400:1089) [ 1186.804671] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1187.315601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1190.965092] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 12:21:13 (1786465273) [ 1192.755544] Lustre: 62059:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c67c51-49cd-4944-9042-40a45bdc0729 at adminstrative request [ 1196.409401] Lustre: Failing over lustre-MDT0000 [ 1196.642477] Lustre: server umount lustre-MDT0000 complete [ 1211.016568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1121) [ 1211.016655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1091 to 0x240000400:1121) [ 1211.050076] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1214.867125] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.423369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1217.872483] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1226.006866] Lustre: DEBUG MARKER: before 4096, after 4096 [ 1228.227300] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 12:21:51 (1786465311) [ 1228.570449] Lustre: 64132:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c67c51-49cd-4944-9042-40a45bdc0729 at adminstrative request [ 1233.707551] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:21:56 (1786465316) [ 1235.063186] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1235.760296] Lustre: Failing over lustre-MDT0000 [ 1235.966333] Lustre: server umount lustre-MDT0000 complete [ 1250.230686] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1252.023763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1123 to 0x240000400:1153) [ 1252.024047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1124 to 0x280000400:1153) [ 1254.312842] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.853986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1258.350352] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:22:21 (1786465341) [ 1259.767665] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1260.483153] Lustre: Failing over lustre-MDT0000 [ 1260.642707] Lustre: server umount lustre-MDT0000 complete [ 1275.165989] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1277.192856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1155 to 0x240000400:1185) [ 1277.192879] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1155 to 0x280000400:1185) [ 1279.226249] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.809618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1283.677768] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:22:46 (1786465366) [ 1285.246590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1286.017672] Lustre: Failing over lustre-MDT0000 [ 1286.247774] Lustre: server umount lustre-MDT0000 complete [ 1301.054869] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1303.209137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1187 to 0x280000400:1217) [ 1303.209150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1187 to 0x240000400:1217) [ 1305.331102] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.830049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.257616] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 12:23:12 (1786465392) [ 1310.564898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1311.202552] Lustre: Failing over lustre-MDT0000 [ 1311.452121] Lustre: server umount lustre-MDT0000 complete [ 1325.979522] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1328.016697] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1249) [ 1328.016704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1219 to 0x240000400:1249) [ 1329.874898] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1330.384510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1333.957493] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:23:36 (1786465416) [ 1335.423033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1336.104149] Lustre: Failing over lustre-MDT0000 [ 1336.361426] Lustre: server umount lustre-MDT0000 complete [ 1350.562363] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1350.933414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1251 to 0x240000400:1281) [ 1350.933476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1281) [ 1354.421104] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.936708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1358.544295] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:24:01 (1786465441) [ 1359.876185] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1360.672367] Lustre: Failing over lustre-MDT0000 [ 1360.893473] Lustre: server umount lustre-MDT0000 complete [ 1373.889134] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1373.891856] Lustre: Skipped 9 previous similar messages [ 1374.884686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1283 to 0x240000400:1313) [ 1374.884938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1283 to 0x280000400:1313) [ 1375.439224] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1379.794123] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.328956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1383.582031] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:24:26 (1786465466) [ 1384.807696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1385.526516] Lustre: Failing over lustre-MDT0000 [ 1385.798441] Lustre: server umount lustre-MDT0000 complete [ 1398.906370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.909188] Lustre: Skipped 19 previous similar messages [ 1400.395836] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1400.499228] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1315 to 0x280000400:1345) [ 1400.499295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1315 to 0x240000400:1345) [ 1404.489130] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.045126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1405.408142] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786465472/real 1786465472] req@ffff9c5c02ee5500 x1873243430018304/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786465488 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1405.417676] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 93 previous similar messages [ 1408.781365] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:24:51 (1786465491) [ 1410.247877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1411.030973] Lustre: Failing over lustre-MDT0000 [ 1411.277310] Lustre: server umount lustre-MDT0000 complete [ 1425.510577] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1426.066101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1347 to 0x240000400:1377) [ 1426.068945] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1347 to 0x280000400:1377) [ 1429.554096] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.105563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1433.746931] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:25:16 (1786465516) [ 1435.232782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1435.990537] Lustre: Failing over lustre-MDT0000 [ 1436.214926] Lustre: server umount lustre-MDT0000 complete [ 1450.480154] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1451.661853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1379 to 0x240000400:1409) [ 1451.662162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1379 to 0x280000400:1409) [ 1454.473887] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1455.043399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1458.835326] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:25:41 (1786465541) [ 1460.189388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1460.861548] Lustre: Failing over lustre-MDT0000 [ 1461.087867] Lustre: server umount lustre-MDT0000 complete [ 1475.856386] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1477.268065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1411 to 0x280000400:1441) [ 1477.268442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1411 to 0x240000400:1441) [ 1480.152332] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1480.762198] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1484.448380] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:26:07 (1786465567) [ 1486.014719] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1486.709152] Lustre: Failing over lustre-MDT0000 [ 1486.912316] Lustre: server umount lustre-MDT0000 complete [ 1501.182234] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1514.085422] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1514.087871] Lustre: Skipped 18 previous similar messages [ 1514.127920] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1514.132199] Lustre: Skipped 18 previous similar messages [ 1514.149421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1443 to 0x240000400:1473) [ 1514.149427] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1443 to 0x280000400:1473) [ 1516.154061] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1516.655874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1520.143514] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:26:43 (1786465603) [ 1520.498874] Lustre: 81751:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c67c51-49cd-4944-9042-40a45bdc0729 at adminstrative request [ 1525.130837] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:26:48 (1786465608) [ 1526.153929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.740911] Lustre: Failing over lustre-MDT0000 [ 1526.972727] Lustre: server umount lustre-MDT0000 complete [ 1529.725890] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.732770] Lustre: Skipped 38 previous similar messages [ 1529.779792] Lustre: lustre-MDT0000: Aborting client recovery [ 1529.781432] LustreError: 82658:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1529.784323] Lustre: 82703:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1529.787351] Lustre: 82703:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 1529.791543] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1529.801788] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1529.833513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1479 to 0x240000400:1505) [ 1529.833519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1480 to 0x280000400:1505) [ 1531.132109] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1534.946402] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1534.949083] Lustre: Skipped 39 previous similar messages [ 1537.148224] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 12:26:59 (1786465619) [ 1538.431673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1539.075476] Lustre: Failing over lustre-MDT0000 [ 1539.273784] Lustre: server umount lustre-MDT0000 complete [ 1542.456741] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1542.473956] Lustre: lustre-MDT0000: Aborting client recovery [ 1542.475993] LustreError: 84021:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1542.479218] Lustre: 84068:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1542.483112] Lustre: 84068:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1542.485422] Lustre: 84068:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 1542.489089] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1542.509617] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1542.551404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1516 to 0x240000400:1537) [ 1542.551943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1516 to 0x280000400:1537) [ 1543.975751] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1547.258818] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1549.456251] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 12:27:12 (1786465632) [ 1550.519599] LustreError: 84771:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1550.523356] LustreError: 84771:0:(osd_handler.c:720:osd_ro()) Skipped 18 previous similar messages [ 1550.849837] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1551.579086] Lustre: Failing over lustre-MDT0000 [ 1551.691580] Lustre: server umount lustre-MDT0000 complete [ 1554.965384] Lustre: lustre-MDT0000: Aborting client recovery [ 1554.966711] LustreError: 85383:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1554.968571] Lustre: 85430:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1554.970389] Lustre: 85430:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1554.973472] Lustre: 85430:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 1554.978829] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1554.998556] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1555.040625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1544 to 0x240000400:1569) [ 1555.040880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1569) [ 1556.443778] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1562.211908] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 12:27:25 (1786465645) [ 1562.600088] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1562.601548] LustreError: 85392:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d365ae680 x1873243423043456/t201863462916(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:511/0 lens 512/456 e 0 to 0 dl 1786465656 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1565.318267] Lustre: Failing over lustre-MDT0000 [ 1565.441968] Lustre: server umount lustre-MDT0000 complete [ 1568.294488] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.298375] LustreError: Skipped 20 previous similar messages [ 1568.453906] Lustre: lustre-MDT0000: Aborting client recovery [ 1568.455705] LustreError: 86592:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1568.458248] Lustre: 86638:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1568.461653] Lustre: 86638:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1568.464708] Lustre: 86638:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 1568.469196] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1568.484754] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1568.522126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1571 to 0x240000400:1601) [ 1568.522566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1601) [ 1569.932142] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1575.695161] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1576.540468] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 12:27:39 (1786465659) [ 1579.202298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1580.792782] Lustre: Failing over lustre-MDT0000 [ 1581.090051] Lustre: server umount lustre-MDT0000 complete [ 1587.407562] Lustre: lustre-MDT0000: Aborting client recovery [ 1587.410728] LustreError: 88053:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1587.414688] Lustre: 88101:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1587.418414] Lustre: 88101:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1587.421428] Lustre: 88101:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 1587.426145] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1587.455411] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1587.535360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1571 to 0x240000400:1633) [ 1587.537067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1633) [ 1590.441532] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1600.447725] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:28:02 (1786465682) [ 1626.451798] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1627.563635] Lustre: Failing over lustre-MDT0000 [ 1627.728598] Lustre: server umount lustre-MDT0000 complete [ 1645.164260] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1647.317174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2034 to 0x240000400:2049) [ 1647.317417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049) [ 1650.692072] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1651.645266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1665.075547] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:29:07 (1786465747) [ 1677.734921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1683.990838] Lustre: Failing over lustre-MDT0000 [ 1684.200740] Lustre: server umount lustre-MDT0000 complete [ 1707.079653] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1708.697429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465) [ 1708.699214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2450 to 0x240000400:2465) [ 1717.608194] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1718.975326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1742.902318] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:30:24 (1786465824) [ 1745.756451] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1746.772659] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1746.779622] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1754.407848] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:30:36 (1786465836) [ 1775.403202] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1785.761555] Lustre: Failing over lustre-OST0000 [ 1785.827271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1785.832448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.898252] Lustre: server umount lustre-OST0000 complete [ 1788.385990] LustreError: 36421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.585705] LustreError: 6683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.510073] LustreError: 8203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.686385] LustreError: 36419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.808627] LustreError: 14795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.821028] LustreError: 14795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1807.974357] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1862.459768] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:32:24 (1786465944) [ 1865.884053] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1868.010728] Lustre: Failing over lustre-MDT0000 [ 1868.244813] Lustre: server umount lustre-MDT0000 complete [ 1887.766869] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1887.771113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2881) [ 1887.845192] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1894.454829] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1895.507949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1903.077416] LustreError: 95972:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1903.084715] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1904.096587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2913) [ 1912.839482] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:33:15 (1786465995) [ 1917.027792] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1922.528148] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1922.538174] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 1922.586364] LustreError: 14795:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 waking [ 1924.254609] LustreError: 95950:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1929.697640] LustreError: 95950:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1929.704930] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 1931.232037] LustreError: 97058:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1936.352357] LustreError: 97058:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1936.357503] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 1937.956942] LustreError: 95949:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1943.008208] LustreError: 95949:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1944.298995] LustreError: 97058:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1949.666183] LustreError: 97058:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1949.674227] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 1949.697813] Lustre: Skipped 1 previous similar message [ 1957.801706] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1957.807746] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1962.976187] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1962.983898] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1969.634346] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 1969.644895] Lustre: Skipped 2 previous similar messages [ 1978.684387] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1978.690851] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1983.969176] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1983.977463] LustreError: 95947:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1991.362714] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:34:33 (1786466073) [ 1993.344478] LustreError: 95947:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2003.561962] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2008.684529] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2013.801162] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2016.965253] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2024.037657] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2024.048305] Lustre: Skipped 1 previous similar message [ 2033.432120] LustreError: 95947:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2033.438200] Lustre: 95947:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9c5d274c1c00 x1873243425673984/t0(0) o38->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:0/0 lens 520/416 e 0 to 0 dl 1786466096 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2034.279525] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 2034.291610] Lustre: Skipped 3 previous similar messages [ 2034.300625] LustreError: 97058:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2058.856192] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2058.865057] Lustre: Skipped 1 previous similar message [ 2074.400145] LustreError: 97058:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2074.415417] Lustre: 97058:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9c5d3e719f80 x1873243425677312/t0(0) o38->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:0/0 lens 520/416 e 0 to 0 dl 1786466137 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2079.336355] LustreError: 95949:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2103.912264] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2103.917921] Lustre: Skipped 4 previous similar messages [ 2119.432157] LustreError: 95949:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2119.445068] Lustre: 95949:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9c5c024c5180 x1873243425679488/t0(0) o38->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:0/0 lens 520/416 e 0 to 0 dl 1786466182 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2124.389936] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 2124.403648] Lustre: Skipped 1 previous similar message [ 2124.405744] LustreError: 97659:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2148.970259] Lustre: lustre-MDT0000: Export ffff9c5d03500000 already connecting from 192.168.201.56@tcp [ 2148.976215] Lustre: Skipped 4 previous similar messages [ 2164.456125] LustreError: 97659:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2164.464593] Lustre: 97659:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9c5d365f6a00 x1873243425681664/t0(0) o38->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:0/0 lens 520/416 e 0 to 0 dl 1786466227 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2169.448833] LustreError: 95947:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2209.520136] LustreError: 95947:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2209.528582] Lustre: 95947:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9c5d3544d500 x1873243425683840/t0(0) o38->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:0/0 lens 520/416 e 0 to 0 dl 1786466272 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2214.515991] LustreError: 97058:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2217.771292] LustreError: 97058:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout interrupted [ 2221.535353] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:38:23 (1786466303) [ 2223.692901] LustreError: 99800:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2223.696647] LustreError: 99800:0:(osd_handler.c:720:osd_ro()) Skipped 5 previous similar messages [ 2224.373830] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2227.679714] Lustre: Failing over lustre-MDT0000 [ 2227.976510] Lustre: server umount lustre-MDT0000 complete [ 2234.336119] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.342350] LustreError: Skipped 4 previous similar messages [ 2234.548301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.548704] Lustre: *** cfs_fail_loc=712, val=0*** [ 2234.558391] Lustre: Skipped 18 previous similar messages [ 2234.567112] LustreError: 6686:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9c5d1e868a80 x1873243430706176/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2234.730044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.732866] Lustre: Skipped 13 previous similar messages [ 2234.789160] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2234.789679] Lustre: lustre-MDT0000: Aborting client recovery [ 2234.794249] Lustre: Skipped 24 previous similar messages [ 2234.799416] LustreError: 100469:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2234.809852] Lustre: 100514:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2234.815353] Lustre: 100514:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 2234.820898] Lustre: 100514:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 00c67c51-49cd-4944-9042-40a45bdc0729@ [ 2234.828259] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2234.865512] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2234.966996] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2913) [ 2234.967423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2945) [ 2238.390419] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2239.970442] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2239.976492] Lustre: Skipped 18 previous similar messages [ 2245.378689] Lustre: Failing over lustre-MDT0000 [ 2245.665188] Lustre: server umount lustre-MDT0000 complete [ 2265.748360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2265.751359] Lustre: Skipped 4 previous similar messages [ 2265.902532] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2265.912581] Lustre: Skipped 4 previous similar messages [ 2265.950895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2867 to 0x240000400:2977) [ 2265.952824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2866 to 0x280000400:2945) [ 2266.397649] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2266.720269] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786466333/real 1786466333] req@ffff9c5d1e86b100 x1873243430715008/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786466349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2266.745490] Lustre: 3305:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 44 previous similar messages [ 2274.013275] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2275.220731] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2282.108411] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:39:24 (1786466364) [ 2282.199251] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 2282.204123] Lustre: Skipped 2 previous similar messages [ 2288.328597] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:39:30 (1786466370) [ 2289.083596] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2289.085715] LustreError: 101425:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d1e46d180 x1873243425739520/t0(0) o700->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:483/0 lens 264/248 e 0 to 0 dl 1786466383 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2308.638217] Lustre: Failing over lustre-MDT0000 [ 2308.911828] Lustre: server umount lustre-MDT0000 complete [ 2327.263683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2979 to 0x240000400:3009) [ 2327.264328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2947 to 0x280000400:2977) [ 2328.464427] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2335.910646] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.103983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2346.290114] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:40:28 (1786466428) [ 2348.133707] Lustre: Failing over lustre-OST0000 [ 2348.259248] Lustre: server umount lustre-OST0000 complete [ 2351.084351] LustreError: 6686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.110825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2369.473531] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2376.808651] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2378.335336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2449.453989] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:42:11 (1786466531) [ 2452.394342] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2454.256181] Lustre: Failing over lustre-MDT0000 [ 2454.588084] Lustre: server umount lustre-MDT0000 complete [ 2472.038421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3030 to 0x240000400:3073) [ 2472.041827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2998 to 0x280000400:3041) [ 2475.113226] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2544.666407] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:43:47 (1786466627) [ 2545.790332] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2545.793637] Lustre: Skipped 2 previous similar messages [ 2555.006930] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:43:57 (1786466637) [ 2557.033115] Lustre: Failing over lustre-MDT0000 [ 2557.282493] Lustre: server umount lustre-MDT0000 complete [ 2572.903924] LustreError: 107856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.920287] LustreError: 107856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2574.716536] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2574.724955] LustreError: 107893:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5c03ae9f80 x1873243425857280/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:13/0 lens 328/344 e 0 to 0 dl 1786466668 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2576.089621] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2591.347725] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 1:24 [ 2591.430814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3085 to 0x240000400:3105) [ 2591.433120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3073) [ 2595.394091] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2596.383203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2603.011050] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:44:45 (1786466685) [ 2604.834520] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2608.124879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2609.489166] Lustre: Failing over lustre-MDT0000 [ 2609.740602] Lustre: server umount lustre-MDT0000 complete [ 2628.933996] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2631.347033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3107 to 0x240000400:3137) [ 2631.350161] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3105) [ 2635.642942] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2636.627880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2642.928358] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:45:25 (1786466725) [ 2643.754106] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2647.877972] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2649.242115] Lustre: Failing over lustre-MDT0000 [ 2649.430923] Lustre: server umount lustre-MDT0000 complete [ 2667.216296] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2670.220666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3137) [ 2670.220675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3169) [ 2673.347794] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2674.198519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2679.276421] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:46:01 (1786466761) [ 2679.929633] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2683.407432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2684.435515] Lustre: Failing over lustre-MDT0000 [ 2684.558443] Lustre: server umount lustre-MDT0000 complete [ 2702.117434] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2706.114530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3201) [ 2706.115619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3169) [ 2712.861368] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:46:35 (1786466795) [ 2714.650828] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2714.656673] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2714.659775] LustreError: 112987:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d3e1e6300 x1873243425903488/t257698037777(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:153/0 lens 392/456 e 0 to 0 dl 1786466808 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2716.396522] Lustre: Failing over lustre-MDT0000 [ 2716.610513] Lustre: server umount lustre-MDT0000 complete [ 2735.491604] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2741.449770] Lustre: 114279:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5c03a1ce00 x1873243425903488/t257698037777(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:180/0 lens 392/456 e 0 to 0 dl 1786466835 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2741.460048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3233) [ 2741.466671] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3171 to 0x280000400:3201) [ 2745.369284] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2746.297177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2752.011109] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:47:14 (1786466834) [ 2752.888603] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2752.890880] LustreError: 114278:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d3fbefb80 x1873243425916800/t261993005072(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:191/0 lens 504/448 e 0 to 0 dl 1786466846 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2756.624787] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2757.848571] Lustre: Failing over lustre-MDT0000 [ 2758.104140] Lustre: server umount lustre-MDT0000 complete [ 2775.494618] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2778.747261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3139 to 0x240000400:3265) [ 2778.749221] Lustre: 115957:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5c019ff100 x1873243425916800/t261993005072(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:217/0 lens 504/2880 e 0 to 0 dl 1786466872 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2778.752492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3233) [ 2781.836536] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2782.670877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2787.501737] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:47:50 (1786466870) [ 2788.129201] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2788.131396] LustreError: 115958:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d1eacc000 x1873243425929728/t266287972368(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:227/0 lens 504/448 e 0 to 0 dl 1786466882 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2789.667861] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2791.718714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2792.821722] Lustre: Failing over lustre-MDT0000 [ 2792.969996] Lustre: server umount lustre-MDT0000 complete [ 2810.568888] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2814.640421] Lustre: 117644:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d3d1aca80 x1873243425930496/t266287972369(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:253/0 lens 392/456 e 0 to 0 dl 1786466908 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2814.654921] Lustre: 117644:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2814.657873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3267 to 0x240000400:3297) [ 2814.657873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3265) [ 2821.577931] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:48:23 (1786466903) [ 2822.232607] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2822.236579] Lustre: Skipped 1 previous similar message [ 2822.240327] LustreError: 117643:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d0610e680 x1873243425942144/t270582939664(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:261/0 lens 504/448 e 0 to 0 dl 1786466916 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2822.261582] LustreError: 117643:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 2823.846134] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2823.848384] Lustre: Skipped 1 previous similar message [ 2826.027587] LustreError: 118579:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2826.031286] LustreError: 118579:0:(osd_handler.c:720:osd_ro()) Skipped 6 previous similar messages [ 2826.560627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2827.804879] Lustre: Failing over lustre-MDT0000 [ 2828.032770] Lustre: server umount lustre-MDT0000 complete [ 2843.709576] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2843.716648] LustreError: Skipped 10 previous similar messages [ 2843.920406] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.926053] Lustre: Skipped 25 previous similar messages [ 2843.999232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.002298] Lustre: Skipped 11 previous similar messages [ 2844.047205] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2844.055718] Lustre: Skipped 13 previous similar messages [ 2846.851262] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2849.016660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3297) [ 2849.016990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3299 to 0x240000400:3329) [ 2849.022417] Lustre: 119176:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d0610f800 x1873243425942144/t270582939664(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:288/0 lens 504/2880 e 0 to 0 dl 1786466943 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2849.252935] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2849.261219] Lustre: Skipped 24 previous similar messages [ 2855.809086] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:48:58 (1786466938) [ 2856.686763] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2858.411729] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2858.425631] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2858.430679] LustreError: 119179:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5c0198b100 x1873243425954816/t274877906960(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:297/0 lens 392/456 e 0 to 0 dl 1786466952 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2861.452180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2862.711742] Lustre: Failing over lustre-MDT0000 [ 2862.938218] Lustre: server umount lustre-MDT0000 complete [ 2880.417832] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786466947/real 1786466947] req@ffff9c5d104d5180 x1873243430918272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786466963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2880.449923] Lustre: 3303:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 69 previous similar messages [ 2881.226714] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2883.176110] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2883.179921] Lustre: Skipped 11 previous similar messages [ 2883.233270] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2883.242912] Lustre: Skipped 11 previous similar messages [ 2883.254717] Lustre: 120610:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d104d5180 x1873243425954816/t274877906960(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:322/0 lens 392/456 e 0 to 0 dl 1786466977 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2883.278435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3331 to 0x240000400:3361) [ 2883.278745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3329) [ 2889.803263] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:49:32 (1786466972) [ 2890.313950] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2905.706861] Lustre: 120608:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d3fc4ce00 x1873243425965696/t279172874255(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:344/0 lens 664/3488 e 0 to 0 dl 1786466999 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2910.027784] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:49:52 (1786466992) [ 2912.139245] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2913.246758] Lustre: Failing over lustre-MDT0000 [ 2913.415355] Lustre: server umount lustre-MDT0000 complete [ 2930.854585] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2931.366916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3363 to 0x240000400:3393) [ 2931.367069] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3203 to 0x280000400:3361) [ 2936.986845] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2938.103859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2952.836946] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:50:35 (1786467035) [ 2955.241610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2956.378613] Lustre: Failing over lustre-MDT0000 [ 2956.645376] Lustre: server umount lustre-MDT0000 complete [ 2972.322419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3363 to 0x240000400:3425) [ 2972.322433] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3363 to 0x280000400:3393) [ 2973.591582] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2979.644617] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2980.672565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2983.877535] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2990.657081] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:51:13 (1786467073) [ 3013.505647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3014.797032] Lustre: Failing over lustre-MDT0000 [ 3015.062545] Lustre: server umount lustre-MDT0000 complete [ 3032.468725] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3033.868683] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4673) [ 3033.868788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4676 to 0x240000400:4705) [ 3038.424701] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3039.427566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3077.943686] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:52:40 (1786467160) [ 3080.748311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3081.982900] Lustre: Failing over lustre-MDT0000 [ 3082.166169] Lustre: server umount lustre-MDT0000 complete [ 3097.191888] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (not set up) [ 3098.683900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4644 to 0x280000400:4705) [ 3098.686097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4707 to 0x240000400:4737) [ 3099.847071] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3106.451515] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3107.357626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3113.622512] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 3114.446910] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3118.123762] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:53:20 (1786467200) [ 3124.119623] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3139.693547] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnecting [ 3139.700601] Lustre: Skipped 2 previous similar messages [ 3141.190371] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3141.199664] Lustre: Skipped 1 previous similar message [ 3141.205293] LustreError: 129134:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5d1849e450 x1873243428740864/t296352743435(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:580/0 lens 66040/440 e 0 to 0 dl 1786467235 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3141.226154] LustreError: 129134:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 3157.625691] Lustre: 129134:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d1e01dc00 x1873243428740864/t296352743435(0) o36->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:596/0 lens 66040/440 e 0 to 0 dl 1786467251 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3163.233839] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3164.036702] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:54:06 (1786467246) [ 3169.513263] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3171.117794] Lustre: Failing over lustre-MDT0000 [ 3171.313874] Lustre: server umount lustre-MDT0000 complete [ 3187.731540] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3188.857216] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4806 to 0x280000400:4833) [ 3188.859572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4839 to 0x240000400:4865) [ 3193.630415] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3194.659219] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3200.244400] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:54:42 (1786467282) [ 3210.072364] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3217.167067] Lustre: Failing over lustre-OST0000 [ 3217.215669] Lustre: server umount lustre-OST0000 complete [ 3219.048508] LustreError: 36411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.493760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3219.498604] LustreError: Skipped 4 previous similar messages [ 3224.167602] LustreError: 36423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.175507] LustreError: 36423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3235.762349] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3248.760651] Lustre: Failing over lustre-OST0000 [ 3248.829176] Lustre: server umount lustre-OST0000 complete [ 3249.634387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.641133] LustreError: Skipped 3 previous similar messages [ 3249.644442] LustreError: 36415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.662888] LustreError: 36415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3266.378778] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3271.840443] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3272.878255] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3308.247918] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:56:30 (1786467390) [ 3309.543268] Lustre: Failing over lustre-MDT0000 [ 3309.773251] Lustre: server umount lustre-MDT0000 complete [ 3326.075318] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3326.603906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5281) [ 3326.603924] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5249) [ 3337.930333] Lustre: Failing over lustre-MDT0000 [ 3338.080452] Lustre: server umount lustre-MDT0000 complete [ 3354.622766] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3357.117338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5234 to 0x280000400:5281) [ 3357.117366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5266 to 0x240000400:5313) [ 3359.942025] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3360.871774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3366.124876] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:57:28 (1786467448) [ 3378.566302] Lustre: Failing over lustre-OST0000 [ 3378.622763] Lustre: server umount lustre-OST0000 complete [ 3382.889215] LustreError: 6685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.900883] LustreError: 6685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3383.264608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.492111] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3402.234968] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3403.162391] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3409.091783] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:58:11 (1786467491) [ 3410.215873] Lustre: Failing over lustre-MDT0000 [ 3410.410178] Lustre: server umount lustre-MDT0000 complete [ 3415.334571] Lustre: *** cfs_fail_loc=605, val=0*** [ 3415.336130] LustreError: 138921:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc1120110 failed: rc = -95 [ 3415.342862] LustreError: 138921:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 3415.348668] LustreError: 138921:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 3415.352927] LustreError: 138921:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 3415.356478] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3415.359606] LustreError: 138921:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3415.409638] Lustre: server umount lustre-MDT0000 complete [ 3415.412691] LustreError: 138921:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3421.972686] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3423.938308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5283 to 0x280000400:5313) [ 3423.938867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5315 to 0x240000400:5345) [ 3426.696184] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:58:29 (1786467509) [ 3428.727708] LustreError: 139962:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3428.735244] LustreError: 139962:0:(osd_handler.c:720:osd_ro()) Skipped 7 previous similar messages [ 3429.313301] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3431.547944] Lustre: Failing over lustre-MDT0000 [ 3431.733684] Lustre: server umount lustre-MDT0000 complete [ 3446.787886] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.792862] LustreError: Skipped 9 previous similar messages [ 3446.981197] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.991141] Lustre: Skipped 22 previous similar messages [ 3447.085533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.088592] Lustre: Skipped 12 previous similar messages [ 3447.138494] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3447.146041] Lustre: Skipped 12 previous similar messages [ 3449.489376] Lustre: *** cfs_fail_loc=707, val=0*** [ 3449.517425] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3452.389184] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3452.399517] Lustre: Skipped 22 previous similar messages [ 3464.833088] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 3465.237080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5327 to 0x280000400:5345) [ 3465.240955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5358 to 0x240000400:5377) [ 3468.897499] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.840801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3475.556768] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:59:18 (1786467558) [ 3501.424262] LustreError: 140614:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d1eacd500 x1873243429604480/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:185/0 lens 664/0 e 0 to 0 dl 1786467595 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3501.433078] LustreError: 140614:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3512.464363] LustreError: 140614:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3512.481505] LustreError: 140617:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d2dfe3100 x1873243429605248/t0(0) o35->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:202/0 lens 392/0 e 0 to 0 dl 1786467612 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3513.755205] LustreError: 140615:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d06fadf80 x1873243429611136/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:236/0 lens 576/0 e 0 to 0 dl 1786467646 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 3513.771221] LustreError: 140615:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 3516.412324] LustreError: 36419:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d365aea00 x1873243431906688/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:200/0 lens 544/0 e 0 to 0 dl 1786467610 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 3516.422063] LustreError: 36419:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 24 previous similar messages [ 3521.532236] LustreError: 36422:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d07f0b100 x1873243431908480/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:205/0 lens 544/0 e 0 to 0 dl 1786467615 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 3521.541024] LustreError: 36422:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 3 previous similar messages [ 3526.064589] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 13:00:08 (1786467608) [ 3550.730182] LustreError: 6691:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3561.768137] LustreError: 6691:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3566.163207] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 13:00:48 (1786467648) [ 3590.207800] LustreError: 142053:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5c09aead80 x1873243429625344/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:274/0 lens 576/0 e 0 to 0 dl 1786467684 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3590.226684] LustreError: 142053:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3595.328143] LustreError: 142053:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3606.560181] LustreError: 140616:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3606.568367] LustreError: 140614:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5c021b3100 x1873243429647232/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:329/0 lens 664/0 e 0 to 0 dl 1786467739 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3606.577186] LustreError: 140614:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 120 previous similar messages [ 3620.423628] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:01:42 (1786467702) [ 3718.274773] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:03:20 (1786467800) [ 3743.304231] LustreError: 140615:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9c5d07f5dc00 x1873243429700736/t0(0) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:427/0 lens 576/0 e 0 to 0 dl 1786467837 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3743.314825] LustreError: 140615:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3743.318855] LustreError: 140615:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3743.323275] LustreError: 140615:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3743.744125] LustreError: 140615:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3759.348269] LustreError: 141029:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3759.350970] LustreError: 141029:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 3759.769499] LustreError: 141029:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3759.773449] LustreError: 141029:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 3789.188460] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:04:31 (1786467871) [ 3816.481639] Lustre: DEBUG MARKER: phase 2 [ 3821.515474] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 13:05:04 (1786467904) [ 3894.743525] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 13:06:17 (1786467977) [ 3895.382959] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3896.288458] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 13:06:18 (1786467978) [ 3898.263952] Lustre: DEBUG MARKER: Started rundbench load pid=128449 ... [ 3901.475487] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3903.278279] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3904.181074] Lustre: Failing over lustre-MDT0000 [ 3904.449899] Lustre: server umount lustre-MDT0000 complete [ 3920.128606] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3924.448214] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786467991/real 1786467991] req@ffff9c5d087ea680 x1873243432035712/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786468007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3924.460271] Lustre: 3302:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 56 previous similar messages [ 3925.605744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3925.609812] Lustre: Skipped 12 previous similar messages [ 3925.961151] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3925.964474] Lustre: Skipped 12 previous similar messages [ 3925.980071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5431 to 0x280000400:5473) [ 3925.980550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5485 to 0x240000400:5505) [ 3928.259334] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3928.979099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3933.485950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3935.174991] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 3936.006403] Lustre: Failing over lustre-MDT0000 [ 3936.192459] Lustre: server umount lustre-MDT0000 complete [ 3951.434351] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3957.063888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5559 to 0x240000400:5601) [ 3957.064255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5526 to 0x280000400:5569) [ 3959.452533] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3960.062896] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3963.951225] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3965.586873] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3966.301747] Lustre: Failing over lustre-MDT0000 [ 3966.533549] Lustre: server umount lustre-MDT0000 complete [ 3981.787816] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3987.551039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5639 to 0x240000400:5665) [ 3987.553283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5607 to 0x280000400:5633) [ 3989.878584] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3990.866440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3995.025925] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3996.681952] Lustre: DEBUG MARKER: test_70b fail mds1 4 times [ 3997.480279] Lustre: Failing over lustre-MDT0000 [ 3997.732841] Lustre: server umount lustre-MDT0000 complete [ 4013.263764] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4016.896731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5709 to 0x240000400:5729) [ 4016.899172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5677 to 0x280000400:5697) [ 4019.094831] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4019.795173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4045.167072] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 13:08:47 (1786468127) [ 4166.824312] LustreError: 153542:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4166.828718] LustreError: 153542:0:(osd_handler.c:720:osd_ro()) Skipped 4 previous similar messages [ 4167.257489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4178.170799] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4178.898036] Lustre: Failing over lustre-MDT0000 [ 4179.211148] Lustre: server umount lustre-MDT0000 complete [ 4192.439409] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.442439] LustreError: Skipped 4 previous similar messages [ 4192.561030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.569496] Lustre: Skipped 9 previous similar messages [ 4192.686625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.689721] Lustre: Skipped 4 previous similar messages [ 4192.732726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4192.735982] Lustre: Skipped 4 previous similar messages [ 4194.349801] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4197.859722] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4197.867349] Lustre: Skipped 9 previous similar messages [ 4200.562790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:7602 to 0x240000400:7617) [ 4200.563049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7570 to 0x280000400:7585) [ 4203.089389] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4203.886106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4327.620417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4338.645591] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4339.466603] Lustre: Failing over lustre-MDT0000 [ 4339.506071] LustreError: 3303:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9c5c0c31dc00 x1873243434914560/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4339.705966] Lustre: server umount lustre-MDT0000 complete [ 4354.403512] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4361.500806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8909 to 0x280000400:8929) [ 4361.501197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:8941 to 0x240000400:8961) [ 4364.077950] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4364.730088] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4380.013672] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 13:14:22 (1786468462) [ 4380.574302] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 4381.193482] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:14:24 (1786468464) [ 4381.838728] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 4382.522733] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:14:25 (1786468465) [ 4387.277737] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4388.886476] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4389.722497] Lustre: Failing over lustre-OST0000 [ 4389.762546] Lustre: server umount lustre-OST0000 complete [ 4392.417076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4392.420715] LustreError: 36421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.426509] LustreError: 36421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4398.694895] LustreError: 149959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.702836] LustreError: 149959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4405.543430] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4409.916234] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4410.585123] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4418.539408] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4420.189964] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 4420.996141] Lustre: Failing over lustre-OST0000 [ 4421.021277] Lustre: server umount lustre-OST0000 complete [ 4423.650192] LustreError: 36413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.662222] LustreError: 36413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4436.925278] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4441.898303] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4442.571435] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4448.642317] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:15:31 (1786468531) [ 4449.251834] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 4449.870615] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:15:32 (1786468532) [ 4451.525751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4452.838110] Lustre: Failing over lustre-MDT0000 [ 4452.976644] Lustre: server umount lustre-MDT0000 complete [ 4467.384504] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4469.328915] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 4484.719132] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 4484.768113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9015 to 0x240000400:9057) [ 4484.768198] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8981 to 0x280000400:9025) [ 4487.079476] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4487.667852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4491.307107] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:16:14 (1786468574) [ 4492.764517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4493.792463] Lustre: Failing over lustre-MDT0000 [ 4494.053703] Lustre: server umount lustre-MDT0000 complete [ 4508.843620] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4510.830915] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4510.832450] LustreError: 162647:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9c5c0ea27b80 x1873243455891584/t347892350979(347892350979) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:439/0 lens 592/608 e 0 to 0 dl 1786468604 ref 1 fl Interpret:/604/0 rc 301/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4526.699820] Lustre: lustre-MDT0000: Client 00c67c51-49cd-4944-9042-40a45bdc0729 (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 4526.705568] Lustre: 162646:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9c5d05112300 x1873243455891584/t347892350979(347892350979) o101->00c67c51-49cd-4944-9042-40a45bdc0729@192.168.201.56@tcp:455/0 lens 592/3488 e 0 to 0 dl 1786468620 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4526.725372] Lustre: lustre-MDT0000: Recovery over after 0:16, of 1 clients 1 recovered and 0 were evicted. [ 4526.731940] Lustre: Skipped 8 previous similar messages [ 4526.747321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9059 to 0x240000400:9089) [ 4526.747548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8981 to 0x280000400:9057) [ 4528.743690] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.276619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4532.824265] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:16:55 (1786468615) [ 4533.867899] Lustre: Failing over lustre-OST0000 [ 4533.907928] Lustre: server umount lustre-OST0000 complete [ 4535.402946] Lustre: Failing over lustre-MDT0000 [ 4535.550402] Lustre: server umount lustre-MDT0000 complete [ 4548.567835] LustreError: 36421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.574385] LustreError: 36421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4548.639360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8981 to 0x280000400:9089) [ 4550.192554] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4552.842403] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4552.845972] Lustre: Skipped 9 previous similar messages [ 4552.847843] Lustre: lustre-OST0000: Denying connection for new client e886093e-bb52-4e0f-b835-068da9e26770 (at 192.168.201.56@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4552.853814] Lustre: Skipped 11 previous similar messages [ 4554.817263] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4554.849153] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786468621/real 1786468621] req@ffff9c5d01492d80 x1873243435272704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786468637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4554.859889] Lustre: 3304:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 42 previous similar messages [ 4557.816765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9059 to 0x240000400:9121) [ 4559.102668] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:17:21 (1786468641) [ 4559.692456] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 4560.360436] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:17:23 (1786468643) [ 4560.983912] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 4561.782554] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:17:24 (1786468644) [ 4562.413167] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 4563.153332] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:17:25 (1786468645) [ 4563.737891] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 4564.315214] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:17:27 (1786468647) [ 4564.869023] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 4565.475727] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:17:28 (1786468648) [ 4566.009746] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 4566.670580] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:17:29 (1786468649) [ 4567.249900] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 4567.811996] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:17:30 (1786468650) [ 4568.363926] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 4569.019978] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:17:31 (1786468651) [ 4569.547049] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 4570.137153] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:17:32 (1786468652) [ 4570.678702] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 4571.221603] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:17:34 (1786468654) [ 4571.822432] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 4572.425018] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:17:35 (1786468655) [ 4572.949577] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 4573.485047] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:17:36 (1786468656) [ 4574.035033] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 4574.631064] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:17:37 (1786468657) [ 4575.214619] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 4575.736690] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:17:38 (1786468658) [ 4576.197838] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 4576.705974] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:17:39 (1786468659) [ 4577.223319] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 4577.749693] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:17:40 (1786468660) [ 4578.357884] Lustre: 167264:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e886093e-bb52-4e0f-b835-068da9e26770 at adminstrative request [ 4582.895750] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:17:45 (1786468665) [ 4584.716732] Lustre: Failing over lustre-MDT0000 [ 4584.942586] Lustre: server umount lustre-MDT0000 complete [ 4599.212131] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4601.389382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9141 to 0x280000400:9185) [ 4601.389382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9173 to 0x240000400:9217) [ 4603.155311] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4603.654454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4606.903956] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:18:09 (1786468689) [ 4610.947815] Lustre: Failing over lustre-OST0000 [ 4612.710830] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 4612.713615] Lustre: Skipped 1 previous similar message [ 4613.053152] Lustre: server umount lustre-OST0000 complete [ 4613.089128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4613.089258] LustreError: 149959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.098372] LustreError: 149959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4627.641237] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4631.209759] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4631.694385] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4635.450622] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:18:38 (1786468718) [ 4636.809217] Lustre: Failing over lustre-MDT0000 [ 4636.929747] Lustre: server umount lustre-MDT0000 complete [ 4639.303944] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9141 to 0x280000400:9217) [ 4639.303960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9318 to 0x240000400:9345) [ 4640.575712] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4643.267833] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:18:46 (1786468726) [ 4644.717641] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4645.533625] Lustre: Failing over lustre-OST0000 [ 4645.559839] Lustre: server umount lustre-OST0000 complete [ 4649.440729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4659.831845] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4663.410439] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4663.887944] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4667.293180] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:19:10 (1786468750) [ 4668.801218] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4670.608762] Lustre: Failing over lustre-OST0000 [ 4670.630379] Lustre: server umount lustre-OST0000 complete [ 4677.221355] LustreError: 36415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.226546] LustreError: 36415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4684.586764] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.56@tcp inode [0x200029c11:0x5:0x0] object 0x240000400:9347 extent [0-1048575]: client csum bf2f2b57, server csum f2e82402 [ 4684.795199] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4688.316529] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4688.785148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4692.025410] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:19:34 (1786468774) [ 4693.265899] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4694.488987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4696.459697] Lustre: Failing over lustre-MDT0000 [ 4696.621759] Lustre: server umount lustre-MDT0000 complete [ 4697.817516] LustreError: 153339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468780 with bad export cookie 18161194241475026294 [ 4707.297275] Lustre: Failing over lustre-OST0000 [ 4707.318277] Lustre: server umount lustre-OST0000 complete [ 4722.144407] LustreError: 3301:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c5d36a05880 x1873243435352704/t0(0) o250->MGC192.168.201.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4723.506936] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4724.391019] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9256 to 0x280000400:9281) [ 4738.329652] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4741.617679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9348 to 0x240000400:9377) [ 4745.372430] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:20:28 (1786468828) [ 4755.316118] Lustre: Failing over lustre-OST0000 [ 4755.357411] Lustre: server umount lustre-OST0000 complete [ 4756.583120] Lustre: Failing over lustre-MDT0000 [ 4756.706769] Lustre: server umount lustre-MDT0000 complete [ 4770.645703] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4771.329551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9256 to 0x280000400:9313) [ 4777.147342] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4777.866152] Lustre: lustre-OST0000: Denying connection for new client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4777.872921] Lustre: Skipped 1 previous similar message [ 4798.565132] Lustre: lustre-OST0000: Denying connection for new client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 4798.571719] Lustre: Skipped 3 previous similar messages [ 4834.405040] Lustre: lustre-OST0000: Denying connection for new client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 4834.411774] Lustre: Skipped 6 previous similar messages [ 4847.500166] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 4847.503595] Lustre: 179480:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client a5d64590-2809-4c08-9207-8ea880dc9a3a@ [ 4847.509939] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 4847.522454] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4847.525611] Lustre: Skipped 21 previous similar messages [ 4847.528353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9388 to 0x240000400:9409) [ 4851.285064] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 4869.292170] Lustre: DEBUG MARKER: free_before: 7520256 free_after: 7520256 [ 4871.282229] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:22:34 (1786468954) [ 4872.618938] Lustre: Failing over lustre-OST0000 [ 4872.650749] Lustre: server umount lustre-OST0000 complete [ 4872.672680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.678269] Lustre: Skipped 18 previous similar messages [ 4872.680103] LustreError: 36419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.686693] LustreError: 36419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 4885.622459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.624169] Lustre: Skipped 16 previous similar messages [ 4885.627992] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4885.630894] Lustre: Skipped 14 previous similar messages [ 4887.646550] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4891.833307] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:22:54 (1786468974) [ 4893.184122] Lustre: Failing over lustre-OST0000 [ 4893.242094] Lustre: server umount lustre-OST0000 complete [ 4907.299163] LustreError: 182755:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 4907.303090] LustreError: 182755:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 81 previous similar messages [ 4907.553769] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4913.120316] Lustre: *** cfs_fail_loc=715, val=40*** [ 4923.492957] Lustre: lustre-OST0000: Client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [ 4929.504128] Lustre: *** cfs_fail_loc=715, val=40*** [ 4929.505795] Lustre: Skipped 1 previous similar message [ 4938.852828] Lustre: lustre-OST0000: Client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp) reconnected, waiting for 2 clients in recovery for 1:08 [ 4938.856996] Lustre: Skipped 1 previous similar message [ 4944.864103] Lustre: *** cfs_fail_loc=715, val=40*** [ 4944.865430] Lustre: Skipped 1 previous similar message [ 4947.344116] LustreError: 182755:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 4947.346488] LustreError: 182755:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 81 previous similar messages [ 4949.227303] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4949.718693] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4952.965732] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:23:55 (1786469035) [ 4954.234779] Lustre: Failing over lustre-MDT0000 [ 4954.449969] Lustre: server umount lustre-MDT0000 complete [ 4967.263539] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.266606] LustreError: Skipped 8 previous similar messages [ 4968.752165] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4969.576792] LustreError: 184352:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 4975.584124] Lustre: *** cfs_fail_loc=715, val=80*** [ 4975.585499] Lustre: Skipped 1 previous similar message [ 4984.932974] Lustre: lustre-MDT0000: Client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 4984.937678] Lustre: Skipped 1 previous similar message [ 4990.944185] Lustre: *** cfs_fail_loc=715, val=80*** [ 5001.316767] Lustre: lustre-MDT0000: Client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:38 [ 5007.328141] Lustre: *** cfs_fail_loc=715, val=80*** [ 5033.064434] Lustre: lustre-MDT0000: Client 5092611b-9240-4458-b7e1-9c6952ddb5ff (at 192.168.201.56@tcp) reconnected, waiting for 1 clients in recovery for 0:06 [ 5033.074557] Lustre: Skipped 1 previous similar message [ 5039.584175] Lustre: *** cfs_fail_loc=715, val=80*** [ 5039.587668] Lustre: Skipped 1 previous similar message [ 5049.445654] Lustre: lustre-MDT0000: Recovery already passed deadline 0:09. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 5049.672147] LustreError: 184352:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 5049.721547] Lustre: 184352:0:(ldlm_lib.c:2951:target_recovery_thread()) too long recovery - read logs [ 5049.731282] LustreError: dumping log to /tmp/lustre-log.1786469132.184352 [ 5049.930964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9423 to 0x240000400:9441) [ 5049.935997] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9326 to 0x280000400:9345) [ 5055.380199] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5056.884272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5063.941483] Lustre: DEBUG MARKER: == replay-single test complete, duration 4937 sec ======== 13:25:46 (1786469146) [ 5065.236926] Lustre: DEBUG MARKER: === replay-single: start cleanup 13:25:47 (1786469147) === [ 5071.716459] Lustre: DEBUG MARKER: === replay-single: finish cleanup 13:25:54 (1786469154) === [ 5072.791585] Lustre: Failing over lustre-MDT0000 [ 5072.976413] Lustre: server umount lustre-MDT0000 complete [ 5088.388980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:9326 to 0x280000400:9377) [ 5088.391667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:9423 to 0x240000400:9473) [ 5089.086613] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5093.766190] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.452484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5103.073317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.076273] Lustre: Skipped 1 previous similar message [ 5107.364325] Lustre: server umount lustre-MDT0000 complete [ 5109.268966] LustreError: 5868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469192 with bad export cookie 18161194241475058662 [ 5109.277744] LustreError: 5868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5109.324914] Lustre: server umount lustre-OST0000 complete [ 5111.232623] Lustre: server umount lustre-OST0001 complete [ 5116.837768] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5118.057698] Key type lgssc unregistered [ 5118.237585] LNet: 187631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5118.240539] LNetError: 187631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5118.258160] LNet: Removed LNI 192.168.201.156@tcp [ 5118.644159] Key type .llcrypt unregistered [ 5118.646055] Key type ._llcrypt unregistered