[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449190943 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002267] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.008434] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011207] LSM: Security Framework initializing [ 0.013105] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.022083] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026790] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] *** VALIDATE tmpfs *** [ 0.030407] *** VALIDATE proc *** [ 0.032085] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034190] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041259] debug: unmapping init [mem 0xffffffff94859000-0xffffffff94860fff] [ 0.044108] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045596] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.048863] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.050972] ... event mask: 000000070000000f [ 0.051274] rcu: Hierarchical SRCU implementation. [ 0.053334] smp: Bringing up secondary CPUs ... [ 0.054535] x86: Booting SMP configuration: [ 0.055041] .... node #0, CPUs: #1 #2 #3 [ 0.058638] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228021] node 0 deferred pages initialised in 164ms [ 0.231100] devtmpfs: initialized [ 0.232262] x86/mm: Memory block size: 128MB [ 0.236730] gcov: version magic: 0x41383552 [ 0.240311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.245318] pinctrl core: initialized pinctrl subsystem [ 0.248212] [ 0.248707] ************************************************************* [ 0.250011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252008] ** ** [ 0.254012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.256011] ** ** [ 0.258011] ** This means that this kernel is built to expose internal ** [ 0.260012] ** IOMMU data structures, which may compromise security on ** [ 0.263015] ** your system. ** [ 0.265009] ** ** [ 0.266009] ** If you see this message and you are not debugging the ** [ 0.268009] ** kernel, report this immediately to your vendor! ** [ 0.269007] ** ** [ 0.270009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272011] ************************************************************* [ 0.274647] NET: Registered protocol family 16 [ 0.277401] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.279054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.281056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.285038] cpuidle: using governor menu [ 0.287074] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.290746] PCI: Using configuration type 1 for base access [ 0.294133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.302132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308260] cryptd: max_cpu_qlen set to 1000 [ 0.310233] ACPI: Added _OSI(Module Device) [ 0.311017] ACPI: Added _OSI(Processor Device) [ 0.312041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.313000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.320105] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.327053] ACPI: Interpreter enabled [ 0.329070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.330024] ACPI: Using IOAPIC for interrupt routing [ 0.332174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.335643] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.346000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.349057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.351022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.354115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359462] acpiphp: Slot [2] registered [ 0.361215] acpiphp: Slot [5] registered [ 0.363223] acpiphp: Slot [6] registered [ 0.365205] acpiphp: Slot [3] registered [ 0.367185] acpiphp: Slot [4] registered [ 0.368137] acpiphp: Slot [7] registered [ 0.370144] acpiphp: Slot [8] registered [ 0.371143] acpiphp: Slot [9] registered [ 0.373134] acpiphp: Slot [10] registered [ 0.374099] acpiphp: Slot [11] registered [ 0.375135] acpiphp: Slot [12] registered [ 0.376124] acpiphp: Slot [13] registered [ 0.378134] acpiphp: Slot [14] registered [ 0.379160] acpiphp: Slot [15] registered [ 0.381147] acpiphp: Slot [16] registered [ 0.382118] acpiphp: Slot [17] registered [ 0.383134] acpiphp: Slot [18] registered [ 0.384116] acpiphp: Slot [19] registered [ 0.385182] acpiphp: Slot [20] registered [ 0.387135] acpiphp: Slot [21] registered [ 0.389145] acpiphp: Slot [22] registered [ 0.390172] acpiphp: Slot [23] registered [ 0.392115] acpiphp: Slot [24] registered [ 0.393116] acpiphp: Slot [25] registered [ 0.394130] acpiphp: Slot [26] registered [ 0.396159] acpiphp: Slot [27] registered [ 0.398129] acpiphp: Slot [28] registered [ 0.399090] acpiphp: Slot [29] registered [ 0.401098] acpiphp: Slot [30] registered [ 0.402120] acpiphp: Slot [31] registered [ 0.404071] PCI host bridge to bus 0000:00 [ 0.405017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.408026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.413033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.418038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420241] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423153] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426467] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.439069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.442047] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.445029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.447027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.451577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.454545] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459706] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.462975] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.473025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.477019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.482016] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.496021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.503019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.538219] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.556016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.572887] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.584747] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.587387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.589330] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.591249] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.593240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599178] iommu: Default domain type: Passthrough [ 0.601517] SCSI subsystem initialized [ 0.603140] ACPI: bus type USB registered [ 0.604101] usbcore: registered new interface driver usbfs [ 0.607080] usbcore: registered new interface driver hub [ 0.609080] usbcore: registered new device driver usb [ 0.611167] pps_core: LinuxPPS API ver. 1 registered [ 0.612009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.614054] PTP clock support registered [ 0.617043] EDAC MC: Ver: 3.0.0 [ 0.618393] PCI: Using ACPI for IRQ routing [ 0.620870] NetLabel: Initializing [ 0.621016] NetLabel: domain hash size = 128 [ 0.623026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626110] NetLabel: unlabeled traffic allowed by default [ 0.628185] vgaarb: loaded [ 0.630309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.632016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.640301] clocksource: Switched to clocksource kvm-clock [ 0.747898] VFS: Disk quotas dquot_6.6.0 [ 0.749300] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.751866] *** VALIDATE ramfs *** [ 0.753262] *** VALIDATE hugetlbfs *** [ 0.755591] pnp: PnP ACPI init [ 0.758126] pnp: PnP ACPI: found 6 devices [ 0.774266] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.776660] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.778596] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.780797] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783443] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.786209] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.789452] NET: Registered protocol family 2 [ 0.792112] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.796758] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.799947] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.805736] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.809631] TCP: Hash tables configured (established 65536 bind 65536) [ 0.813042] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.816431] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.819355] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.821981] NET: Registered protocol family 1 [ 0.824482] RPC: Registered named UNIX socket transport module. [ 0.826563] RPC: Registered udp transport module. [ 0.828311] RPC: Registered tcp transport module. [ 0.830212] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.832869] NET: Registered protocol family 44 [ 0.834806] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.837607] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.839538] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.841843] PCI: CLS 0 bytes, default 64 [ 0.844114] Unpacking initramfs... [ 2.272303] debug: unmapping init [mem 0xffff8c8e3cc64000-0xffff8c8e3ffcffff] [ 2.277186] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.279671] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.282926] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.787951] Initialise system trusted keyrings [ 2.789657] Key type blacklist registered [ 2.791118] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.802213] zbud: loaded [ 2.805946] *** VALIDATE nfs *** [ 2.807599] *** VALIDATE nfs4 *** [ 2.810341] pstore: using deflate compression [ 2.813715] Platform Keyring initialized [ 2.929434] NET: Registered protocol family 38 [ 2.931473] Key type asymmetric registered [ 2.932814] Asymmetric key parser 'x509' registered [ 2.934716] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.937351] io scheduler mq-deadline registered [ 2.939746] io scheduler kyber registered [ 2.941744] io scheduler bfq registered [ 2.945233] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.948486] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.951457] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.954583] ACPI: Power Button [PWRF] [ 2.959899] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.967222] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.980737] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.011535] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.042487] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.049586] Non-volatile memory driver v1.3 [ 3.051152] Linux agpgart interface v0.103 [ 3.081256] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.083628] vda: detected capacity change from 0 to 76791808 [ 3.098125] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.101512] vdb: detected capacity change from 0 to 1073741824 [ 3.118958] libphy: Fixed MDIO Bus: probed [ 3.130496] usbcore: registered new interface driver usbserial_generic [ 3.133240] usbserial: USB Serial support registered for generic [ 3.136723] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.141532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.144116] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.146949] mousedev: PS/2 mouse device common for all mice [ 3.150048] rtc_cmos 00:05: RTC can wake from S4 [ 3.153897] rtc_cmos 00:05: registered as rtc0 [ 3.153904] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155242] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.160326] intel_pstate: CPU model not supported [ 3.163163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.167122] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.167184] hid: raw HID events driver (C) Jiri Kosina [ 3.171672] usbcore: registered new interface driver usbhid [ 3.173754] usbhid: USB HID core driver [ 3.175606] drop_monitor: Initializing network drop monitor service [ 3.178220] Initializing XFRM netlink socket [ 3.180314] NET: Registered protocol family 10 [ 3.182952] Segment Routing with IPv6 [ 3.184522] NET: Registered protocol family 17 [ 3.186585] mpls_gso: MPLS GSO support [ 3.193181] RAS: Correctable Errors collector initialized. [ 3.195549] AVX version of gcm_enc/dec engaged. [ 3.197526] AES CTR mode by8 optimization enabled [ 3.290609] sched_clock: Marking stable (3290552754, 0)->(4204433328, -913880574) [ 3.296127] registered taskstats version 1 [ 3.297821] Loading compiled-in X.509 certificates [ 3.300434] zswap: loaded using pool lzo/zbud [ 3.328212] Key type big_key registered [ 3.342856] Key type encrypted registered [ 3.345427] ima: No TPM chip found, activating TPM-bypass! [ 3.347463] ima: Allocated hash algorithm: sha1 [ 3.348953] ima: No architecture policies found [ 3.350665] evm: Initialising EVM extended attributes: [ 3.352505] evm: security.selinux [ 3.353691] evm: security.ima [ 3.354591] evm: security.capability [ 3.355858] evm: HMAC attrs: 0x1 [ 3.358195] rtc_cmos 00:05: setting system clock to 2026-09-10 14:10:34 UTC (1789049434) [ 3.364394] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 3.366939] debug: unmapping init [mem 0xffffffff94582000-0xffffffff94858fff] [ 3.375136] Write protecting the kernel read-only data: 28672k [ 3.379655] debug: unmapping init [mem 0xffffffff92c03000-0xffffffff92dfffff] [ 3.382556] debug: unmapping init [mem 0xffffffff93514000-0xffffffff935fffff] [ 3.416924] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.427086] systemd[1]: Detected virtualization kvm. [ 3.428672] systemd[1]: Detected architecture x86-64. [ 3.430326] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.454861] systemd[1]: No hostname configured. [ 3.456752] systemd[1]: Set hostname to . [ 3.458699] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460725] systemd[1]: Initializing machine ID from random generator. [ 3.517533] random: ln: uninitialized urandom read (6 bytes read) [ 3.613149] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615679] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.619771] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.624121] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.385994] device-mapper: uevent: version 1.0.3 [ 4.388964] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.204087] virtio_net virtio0 ens2: renamed from eth0 [ 5.320049] scsi host0: ata_piix [ 5.365531] scsi host1: ata_piix [ 5.367281] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.372955] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.646235] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.860709] random: crng init done [ 9.865720] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.374741] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.530903] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.797142] SELinux: Disabled at runtime. [ 11.858768] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.868273] systemd[1]: Detected virtualization kvm. [ 11.870285] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.381603] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.385676] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.395416] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.400429] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.404944] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.412511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.417640] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 12.485501] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.882744] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.169497] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.220325] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.274820] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.303995] EDAC sbridge: Ver: 1.1.2 [ 14.681665] Key type dns_resolver registered [ 14.993888] NFS: Registering the id_resolver key type [ 14.995946] Key type id_resolver registered [ 14.997197] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg647-client login: [ 68.748029] libcfs: loading out-of-tree module taints kernel. [ 68.894521] Key type ._llcrypt registered [ 68.900188] Key type .llcrypt registered [ 69.833304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.858847] alg: No test for adler32 (adler32-zlib) [ 71.474357] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 72.430446] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 74.170186] Key type lgssc registered [ 76.209061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 202.283234] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 207.232770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.898741] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.143329] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing yml_node [ 227.810601] Lustre: lustre-OST0000-osc-ffff8c8e88abc800: disconnect after 23s idle [ 228.443100] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 230.674852] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 232.606447] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 234.012401] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Sep 10 10:14:23 EDT 2026 [ 249.113325] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 250.650314] Lustre: DEBUG MARKER: === replay-single: start setup 10:14:40 (1789049680) === [ 255.841352] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing check_config_client /mnt/lustre [ 272.922804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.128743] Lustre: DEBUG MARKER: === replay-single: finish setup 10:15:13 (1789049713) === [ 284.818504] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 10:15:14 (1789049714) [ 288.597892] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 309.744228] Lustre: lustre-OST0000-osc-ffff8c8e88abc800: disconnect after 24s idle [ 309.755346] Lustre: Skipped 1 previous similar message [ 309.781262] Lustre: lustre-MDT0000-mdc-ffff8c8e88abc800: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 309.827145] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 309.889156] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee62f0 to 0x4c7c0a774ee65f2 [ 309.906341] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 310.624138] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049725/real 1789049725] req@ffff8c8e8a158a80 x1875954371672320/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e88abc800@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789049741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 315.872309] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049730/real 1789049730] req@ffff8c8e8a15b480 x1875954371672832/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e88abc800@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789049746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 320.032388] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049735/real 1789049735] req@ffff8c8e91d70380 x1875954371673344/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e88abc800@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789049751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 324.012914] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 325.284876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 330.410070] hrtimer: interrupt took 3268812 ns [ 334.939763] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 10:16:04 (1789049764) [ 340.454398] Lustre: lustre-OST0000-osc-ffff8c8e88abc800: Connection to lustre-OST0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 350.688214] Lustre: lustre-OST0001-osc-ffff8c8e88abc800: disconnect after 23s idle [ 350.697746] Lustre: Skipped 1 previous similar message [ 358.431693] Lustre: lustre-OST0000-osc-ffff8c8e88abc800: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 358.439756] Lustre: Skipped 1 previous similar message [ 370.712874] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 372.102525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 380.393034] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 10:16:50 (1789049810) [ 384.364979] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 384.487118] Lustre: Unmounted lustre-client [ 411.246508] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 416.252906] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 421.370858] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 426.503742] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 431.606674] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 441.867478] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 441.871676] LustreError: Skipped 1 previous similar message [ 462.329775] LustreError: lustre-MDT0000-mdc-ffff8c8e894f3800: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 462.332839] LustreError: Skipped 3 previous similar messages [ 472.627801] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 481.374233] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 10:18:31 (1789049911) [ 485.080821] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 485.175732] Lustre: Unmounted lustre-client [ 512.387577] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_connect to node 192.168.206.147@tcp failed: rc = -16 [ 512.396579] LustreError: Skipped 1 previous similar message [ 574.054930] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 581.622125] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 10:20:11 (1789050011) [ 584.937389] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 604.662824] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 604.680211] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee74fc to 0x4c7c0a774ee778e [ 604.698430] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 605.664148] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050020/real 1789050020] req@ffff8c8e82e5c000 x1875954371727616/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 605.687965] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 606.380498] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 610.592144] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050025/real 1789050025] req@ffff8c8e91c5b100 x1875954371728128/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.982252] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 620.000284] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050035/real 1789050035] req@ffff8c8e82e5ce00 x1875954371729152/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.029388] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 625.410232] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 626.721740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 633.781733] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 10:21:03 (1789050063) [ 637.215423] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 656.672128] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050071/real 1789050071] req@ffff8c8e895f1880 x1875954371737856/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789050087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 656.698421] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 657.761422] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 660.903813] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e895f3800 x1875954371736960/t21474836484(21474836484) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789050107 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 661.008630] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 666.088071] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee778e to 0x4c7c0a774ee7f91 [ 672.088869] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 673.647891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 681.268922] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 10:21:51 (1789050111) [ 685.446369] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 707.040361] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050122/real 1789050122] req@ffff8c8e82e5c700 x1875954371749248/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050138 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 707.049049] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 707.075552] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 707.084775] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 707.092771] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee7f91 to 0x4c7c0a774ee8269 [ 707.110797] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 707.116330] Lustre: Skipped 1 previous similar message [ 707.169256] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e91c59880 x1875954371747712/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789050154 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 718.425092] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 720.616981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 729.174772] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 10:22:38 (1789050158) [ 732.890411] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 752.036196] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 752.036292] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 752.075816] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee8269 to 0x4c7c0a774ee85b1 [ 752.089899] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 752.098942] Lustre: Skipped 1 previous similar message [ 753.097252] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e852ac700 x1875954371758336/t30064771076(30064771076) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 536/608 e 0 to 0 dl 1789050200 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 764.004507] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 765.506566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 772.486617] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 10:23:22 (1789050202) [ 776.102455] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 798.176455] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050213/real 1789050213] req@ffff8c8e852ad880 x1875954371768576/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789050229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 798.219017] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 798.223218] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 798.254101] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 798.276277] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee85b1 to 0x4c7c0a774ee8ba0 [ 798.289930] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 798.302577] Lustre: Skipped 1 previous similar message [ 809.903547] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 811.612819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 819.791070] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 10:24:09 (1789050249) [ 825.214117] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 842.208686] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 845.280180] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 845.303104] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee8ba0 to 0x4c7c0a774ee8e5c [ 847.210130] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 867.918202] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 867.930679] Lustre: Skipped 2 previous similar messages [ 873.673875] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.163263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 883.631836] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 10:25:13 (1789050313) [ 887.389814] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 908.772482] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 908.799826] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 908.813886] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee8e5c to 0x4c7c0a774ee93aa [ 920.342354] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 921.763914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 929.228329] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 10:25:58 (1789050358) [ 932.822356] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 955.873897] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050370/real 1789050370] req@ffff8c8e852ace00 x1875954371798656/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 955.895620] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 966.566336] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 968.380392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 976.303904] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 10:26:46 (1789050406) [ 980.281174] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1001.760258] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 1001.772544] LustreError: Skipped 1 previous similar message [ 1011.186322] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ee9826 to 0x4c7c0a774ee9d04 [ 1011.207862] Lustre: Skipped 1 previous similar message [ 1011.220380] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 1011.232094] Lustre: Skipped 4 previous similar messages [ 1017.044539] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.469821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1025.700908] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 10:27:35 (1789050455) [ 1029.177618] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1047.008302] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1047.029602] Lustre: Skipped 2 previous similar messages [ 1057.309779] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e82e5d500 x1875954371816320/t55834574851(55834574851) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789050504 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1062.261676] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1063.743929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1073.217615] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 10:28:22 (1789050502) [ 1078.069499] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1096.164298] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_batch to node 192.168.206.147@tcp failed: rc = -107 [ 1096.173402] LustreError: Skipped 11 previous similar messages [ 1111.240243] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1112.692900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1120.743609] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 10:29:10 (1789050550) [ 1124.457029] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1149.924188] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 1149.937518] LustreError: Skipped 2 previous similar messages [ 1160.166247] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774eeb067 to 0x4c7c0a774eee167 [ 1160.172307] Lustre: Skipped 2 previous similar messages [ 1175.594172] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e852ad880 x1875954371866496/t64424509443(64424509443) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789050622 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1175.636622] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 9 previous similar messages [ 1184.138381] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.419088] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1206.023794] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 10:30:35 (1789050635) [ 1209.355665] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1243.356267] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1244.919661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1253.479524] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 10:31:23 (1789050683) [ 1256.539839] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1277.954227] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 1277.960455] Lustre: Skipped 9 previous similar messages [ 1278.752531] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050693/real 1789050693] req@ffff8c8e87d69c00 x1875954372347776/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789050709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1278.809879] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 1291.145785] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1292.591847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1301.268658] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 10:32:11 (1789050731) [ 1304.878120] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1324.002599] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1324.016205] Lustre: Skipped 4 previous similar messages [ 1337.281642] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1339.040722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1347.754694] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 10:32:57 (1789050777) [ 1352.002450] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1395.823305] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87c76300 x1875954372367744/t81604378634(81604378634) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 664/608 e 0 to 0 dl 1789050842 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 1395.876903] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 219 previous similar messages [ 1403.360372] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.077561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1412.897685] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 10:34:02 (1789050842) [ 1416.491564] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1437.664417] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 1437.679343] LustreError: Skipped 4 previous similar messages [ 1437.690154] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ef5454 to 0x4c7c0a774ef58ec [ 1437.710539] Lustre: Skipped 4 previous similar messages [ 1439.276146] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1461.576972] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.205874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1471.393989] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 10:35:01 (1789050901) [ 1475.730844] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1510.406937] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.582472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1518.918917] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 10:35:48 (1789050948) [ 1522.640625] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1544.021769] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87c74e00 x1875954372401664/t94489280523(94489280523) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789050991 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1556.321656] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1557.977819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1565.728217] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 10:36:35 (1789050995) [ 1569.474475] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1590.323970] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87ca0000 x1875954372402432/t94489280526(94489280526) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789051037 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 1590.360897] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 1 previous similar message [ 1605.431840] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1607.328900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1616.344487] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 10:37:26 (1789051046) [ 1620.010888] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1653.796422] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1655.366767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1663.171552] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 10:38:12 (1789051092) [ 1667.186722] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1698.828834] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87ca0000 x1875954372402432/t94489280526(94489280526) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789051145 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 1698.864947] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 3 previous similar messages [ 1710.464907] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1712.009385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1719.384318] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 10:39:09 (1789051149) [ 1723.365386] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1745.899863] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1766.262632] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1767.624139] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1774.561925] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 10:40:04 (1789051204) [ 1778.566839] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1796.064277] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789051211/real 1789051211] req@ffff8c8e87ca2d80 x1875954372461824/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789051227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1796.089244] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 38 previous similar messages [ 1805.543072] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 1805.554278] Lustre: Skipped 19 previous similar messages [ 1811.722441] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1813.164304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1819.992348] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 10:40:49 (1789051249) [ 1824.041585] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1843.686114] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1843.697416] Lustre: Skipped 9 previous similar messages [ 1868.331656] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87ca0000 x1875954372402432/t94489280526(94489280526) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789051315 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 1868.372669] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 5 previous similar messages [ 1874.635925] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.434080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1885.303773] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 10:41:54 (1789051314) [ 1890.262232] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1940.415244] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1941.729331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1949.705945] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 10:42:59 (1789051379) [ 1953.066444] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1971.680160] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 1971.689440] LustreError: Skipped 9 previous similar messages [ 1980.906672] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774ef85c4 to 0x4c7c0a774ef8c00 [ 1980.918098] Lustre: Skipped 9 previous similar messages [ 2000.998984] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2002.480454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2010.623592] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:44:00 (1789051440) [ 2014.262626] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2062.185493] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.626727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2070.499912] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 10:45:00 (1789051500) [ 2076.753631] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_statfs to node 192.168.206.147@tcp failed: rc = -107 [ 2076.771206] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2076.791847] LustreError: 54757:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c8e852a2000: inode [0x200001b71:0x118:0x0] mdc close failed: rc = -108 [ 2076.791925] LustreError: 54701:0:(vvp_io.c:1889:vvp_io_init()) lustre: refresh file layout [0x200001b71:0x132:0x0] error -108. [ 2101.096974] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2125.611308] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.461450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2146.122739] Lustre: DEBUG MARKER: before 4096, after 4096 [ 2151.851607] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 10:46:21 (1789051581) [ 2154.316115] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_statfs to node 192.168.206.147@tcp failed: rc = -107 [ 2154.327127] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2164.079595] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:46:33 (1789051593) [ 2168.220079] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2190.910932] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e87ca2680 x1875954372544128/t141733920773(141733920773) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789051637 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 2190.931633] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 8 previous similar messages [ 2203.955028] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2205.614585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2214.623299] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:47:24 (1789051644) [ 2218.712265] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2254.467511] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.230444] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2264.026704] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 10:48:13 (1789051693) [ 2268.024898] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2312.192092] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.686254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2321.583613] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 10:49:11 (1789051751) [ 2325.640209] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2361.363795] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.719562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2371.997574] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 10:50:02 (1789051802) [ 2375.475305] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2397.664141] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789051812/real 1789051812] req@ffff8c8e87ca2d80 x1875954372592384/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789051828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2397.698416] Lustre: 2353:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 2407.136493] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2408.966879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2419.527987] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:50:48 (1789051848) [ 2423.642035] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2443.744646] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2443.773325] Lustre: Skipped 11 previous similar messages [ 2443.787777] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 2443.805686] Lustre: Skipped 23 previous similar messages [ 2457.039772] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2458.537647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2467.335244] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:51:37 (1789051897) [ 2470.920612] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2519.937482] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2521.169674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2528.394910] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 10:52:38 (1789051958) [ 2531.641756] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2565.199740] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2566.643594] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2575.019864] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:53:24 (1789052004) [ 2579.439663] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2602.474352] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 2602.499492] LustreError: Skipped 10 previous similar messages [ 2602.526673] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774efbf30 to 0x4c7c0a774efc3eb [ 2602.547181] Lustre: Skipped 10 previous similar messages [ 2617.456340] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.993119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2626.101842] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 10:54:16 (1789052056) [ 2630.050377] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2665.322921] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2666.838331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2674.316115] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 10:55:04 (1789052104) [ 2678.698611] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2714.580656] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2716.437565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2724.892529] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 10:55:54 (1789052154) [ 2726.387450] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2735.521917] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 10:56:04 (1789052164) [ 2739.092455] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2752.062094] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2766.780176] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 10:56:36 (1789052196) [ 2771.666755] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2787.890612] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2805.248077] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 10:57:14 (1789052234) [ 2809.296523] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2823.749893] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2834.659597] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 10:57:44 (1789052264) [ 2849.324368] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2864.410555] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2865.797356] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 10:58:15 (1789052295) [ 2869.319103] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2880.035150] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2893.281904] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 10:58:43 (1789052323) [ 2918.424755] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2938.672120] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2961.484489] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2962.734709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2982.622715] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:00:12 (1789052412) [ 3001.368953] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3030.504274] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052445/real 1789052445] req@ffff8c8e9190aa00 x1875954374321920/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789052461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3030.526798] Lustre: 2356:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 28 previous similar messages [ 3041.097684] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3042.444703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3061.270950] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:01:31 (1789052491) [ 3069.401378] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:01:39 (1789052499) [ 3088.288592] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 3101.167471] Lustre: lustre-OST0000-osc-ffff8c8e852a2000: Connection to lustre-OST0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3101.183644] Lustre: Skipped 13 previous similar messages [ 3176.045531] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:03:26 (1789052606) [ 3179.765981] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3199.490178] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 3199.502937] Lustre: Skipped 26 previous similar messages [ 3212.406973] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3214.090398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3232.838842] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:04:22 (1789052662) [ 3312.862214] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:05:42 (1789052742) [ 3334.633967] LustreError: 87123:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3356.711817] LustreError: 87145:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3379.181665] LustreError: 87167:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3401.769817] LustreError: 87190:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3424.245251] LustreError: 87213:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3446.704928] LustreError: 87236:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3469.805963] LustreError: 87258:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3514.354781] LustreError: 87304:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8c8e852a2000: can't stat MDS #0: rc = -114 [ 3514.370431] LustreError: 87304:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 1 previous similar message [ 3543.126843] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:09:32 (1789052972) [ 3547.440625] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3560.439467] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 3560.467363] LustreError: Skipped 10 previous similar messages [ 3560.488822] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774f416bd to 0x4c7c0a774f4281a [ 3560.506673] Lustre: Skipped 10 previous similar messages [ 3560.540884] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3603.607855] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3604.956525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3613.409319] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 11:10:43 (1789053043) [ 3613.902812] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 3613.926689] LustreError: 89682:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8c8e852a2000: inode [0x20001a9e1:0x1:0x0] mdc close failed: rc = -108 [ 3613.976255] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3623.528809] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:10:52 (1789053052) [ 3640.800138] Lustre: 90270:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053055/real 1789053055] req@ffff8c8e91803800 x1875954375456640/t0(0) o700->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:30/10 lens 264/248 e 0 to 1 dl 1789053071 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3640.839124] Lustre: 90270:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 3691.637540] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3692.907229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3702.306614] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:12:12 (1789053132) [ 3707.366757] Lustre: lustre-OST0000-osc-ffff8c8e852a2000: Connection to lustre-OST0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3707.375332] Lustre: Skipped 18 previous similar messages [ 3736.607703] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3738.112300] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3809.751348] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:13:59 (1789053239) [ 3813.541706] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3835.380070] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 3835.386078] Lustre: Skipped 21 previous similar messages [ 3836.531790] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e90db6300 x1875954375516416/t236223201383(236223201383) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789053283 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 [ 3836.566890] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 19 previous similar messages [ 3912.693156] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 11:15:42 (1789053342) [ 3926.555856] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 11:15:56 (1789053356) [ 3969.056679] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3970.460902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3978.629495] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 11:16:48 (1789053408) [ 3985.784417] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4027.688624] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4029.799891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4039.713446] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 11:17:49 (1789053469) [ 4045.288785] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4087.827508] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4089.061422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4096.849511] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 11:18:46 (1789053526) [ 4102.809270] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4139.165690] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 11:19:28 (1789053568) [ 4161.312468] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 4161.320100] LustreError: Skipped 7 previous similar messages [ 4172.377057] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774f465a7 to 0x4c7c0a774f46dfe [ 4172.396360] Lustre: Skipped 7 previous similar messages [ 4177.460289] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4179.280786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4189.242491] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 11:20:17 (1789053617) [ 4195.681704] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4237.588696] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4239.750562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4249.614170] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 11:21:19 (1789053679) [ 4256.647409] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4267.491606] Lustre: 102671:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053682/real 1789053682] req@ffff8c8e90db5880 x1875954375655296/t0(0) o36->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 504/576 e 0 to 1 dl 1789053698 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4267.544433] Lustre: 102671:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 35 previous similar messages [ 4304.904268] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 11:22:14 (1789053734) [ 4313.137832] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4322.784177] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4322.815593] Lustre: Skipped 8 previous similar messages [ 4356.458888] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 11:23:05 (1789053785) [ 4364.819151] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4407.569770] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 11:23:57 (1789053837) [ 4430.167755] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 11:24:20 (1789053860) [ 4434.236303] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4466.665443] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 4466.686020] Lustre: Skipped 20 previous similar messages [ 4477.447480] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4479.228237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4496.933792] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 11:25:26 (1789053926) [ 4501.730523] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4537.573034] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4538.870603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4558.183756] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 11:26:27 (1789053987) [ 4601.825352] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4634.371148] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4635.919186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4686.328628] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 11:28:36 (1789054116) [ 4686.840374] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 4690.708590] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4707.812675] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_batch to node 192.168.206.147@tcp failed: rc = -107 [ 4725.983925] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4727.528987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4731.803987] Lustre: Unmounted lustre-client [ 4736.588879] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 4738.001758] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4743.456725] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 11:29:33 (1789054173) [ 4748.996375] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 4787.212852] Lustre: Unmounted lustre-client [ 4792.464831] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4793.874688] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 11:30:23 (1789054223) [ 4802.104260] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4820.768459] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 4820.777765] LustreError: Skipped 8 previous similar messages [ 4829.999292] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774f89a73 to 0x4c7c0a774f8c775 [ 4830.008748] Lustre: Skipped 8 previous similar messages [ 4831.209548] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 4851.306568] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4852.702876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4861.981820] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 11:31:31 (1789054291) [ 4880.257875] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 4967.262084] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4968.971769] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5009.511849] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 11:33:59 (1789054439) [ 5029.856139] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789054444/real 1789054444] req@ffff8c8ea0f54e00 x1875954379286016/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789054460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5029.898085] Lustre: 2355:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 37 previous similar messages [ 5029.911863] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5029.955043] Lustre: Skipped 11 previous similar messages [ 5070.852033] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 5070.866622] Lustre: Skipped 14 previous similar messages [ 5083.917980] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5085.589807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5094.448272] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 11:35:24 (1789054524) [ 5141.943773] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.545553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5153.133397] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 11:36:23 (1789054583) [ 5181.946478] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 11:36:51 (1789054611) [ 5185.081232] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5230.560248] LustreError: 2352:0:(client.c:3399:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff8c8ea0f54a80 x1875954379311232/t313532612612(313532612612) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/664 e 0 to 1 dl 1789054661 ref 2 fl Interpret:EXQU/604/ffffffff rc -110/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5230.627444] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8ea0f54a80 x1875954379311232/t313532612612(313532612612) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789054677 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 [ 5230.650240] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 19 previous similar messages [ 5236.435770] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5237.664963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5245.567755] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 11:37:55 (1789054675) [ 5301.294895] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 11:38:51 (1789054731) [ 5348.654866] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 11:39:38 (1789054778) [ 5412.449942] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 11:40:42 (1789054842) [ 5439.089494] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5449.112129] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5449.125895] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5459.224114] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5459.252129] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5469.352132] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5469.383395] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5479.401271] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5479.423684] LustreError: 2353:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5489.520125] LustreError: 2353:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5489.536511] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 5499.552114] LustreError: 125406:0:(client.c:1695:after_reply()) cfs_fail_timeout id 50c awake [ 5506.295679] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 11:42:16 (1789054936) [ 5583.693341] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 11:43:33 (1789055013) [ 5615.307178] Lustre: DEBUG MARKER: phase 2 [ 5623.738786] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 11:44:13 (1789055053) [ 5652.174150] LustreError: 31384:0:(ldlm_request.c:1533:ldlm_cli_cancel_req()) cfs_fail_timeout id 312 sleeping for 19000ms [ 5671.240078] LustreError: 31384:0:(ldlm_request.c:1533:ldlm_cli_cancel_req()) cfs_fail_timeout id 312 awake [ 5703.921171] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 11:45:33 (1789055133) [ 5705.459751] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5707.084298] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 11:45:36 (1789055136) [ 5711.376371] Lustre: DEBUG MARKER: Started rundbench load pid=128687 ... [ 5717.020400] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5719.588504] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5721.700349] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation ldlm_enqueue to node 192.168.206.147@tcp failed: rc = -19 [ 5721.713983] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5721.736683] Lustre: Skipped 4 previous similar messages [ 5741.542828] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 5741.568959] LustreError: Skipped 4 previous similar messages [ 5741.598184] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a774f9e74e to 0x4c7c0a774fa48fa [ 5741.606659] Lustre: Skipped 4 previous similar messages [ 5741.622442] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e91961c00 x1875954379507968/t317827580304(317827580304) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789055188 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 [ 5741.640476] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 5741.646902] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 24 previous similar messages [ 5741.665590] Lustre: Skipped 5 previous similar messages [ 5754.071510] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5755.689329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5763.461673] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5765.841613] Lustre: DEBUG MARKER: test_70b fail mds1 2 times [ 5767.766399] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation ldlm_enqueue to node 192.168.206.147@tcp failed: rc = -19 [ 5789.225612] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 5807.095749] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e91961c00 x1875954379507968/t317827580304(317827580304) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789055254 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 [ 5807.115158] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 28 previous similar messages [ 5812.683640] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5814.183631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5822.062730] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5824.638981] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5826.215560] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_close to node 192.168.206.147@tcp failed: rc = -19 [ 5843.552141] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055258/real 1789055258] req@ffff8c8e92002a00 x1875954380486784/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789055274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5843.575886] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 5862.024614] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e91961c00 x1875954379507968/t317827580304(317827580304) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789055309 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 [ 5862.042084] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 39 previous similar messages [ 5867.828647] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5869.098358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5877.713964] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 11:48:27 (1789055307) [ 6003.306696] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6014.938194] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6017.680915] LustreError: lustre-MDT0000-mdc-ffff8c8e852a2000: operation mds_reint to node 192.168.206.147@tcp failed: rc = -19 [ 6038.375983] Lustre: 14134:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 6055.983333] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8c8e87d73100 x1875954384791552/t330712490904(330712490904) o35->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:23/10 lens 392/456 e 0 to 0 dl 1789055503 ref 2 fl Interpret:RQU/604/0 rc -116/-116 job:'tar.0' uid:0 gid:0 projid:0 [ 6055.998086] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 39 previous similar messages [ 6066.502966] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6068.007146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6133.552500] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 11:52:43 (1789055563) [ 6134.744388] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs [ 6136.470453] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 11:52:46 (1789055566) [ 6137.797341] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs [ 6139.592286] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 11:52:49 (1789055569) [ 6147.304952] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6149.516694] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6184.977038] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6186.636513] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6197.175472] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6199.423582] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6232.918163] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6234.538547] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6244.896595] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 11:54:34 (1789055674) [ 6246.272935] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs [ 6247.682279] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 11:54:37 (1789055677) [ 6251.233355] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6310.880183] LustreError: 2352:0:(client.c:3399:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff8c8e865ead80 x1875954387708928/t335007454205(335007454205) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/664 e 0 to 1 dl 1789055742 ref 2 fl Interpret:EXPQU/604/ffffffff rc -110/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 6311.016574] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e865ead80 x1875954387708928/t335007454205(335007454205) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/608 e 0 to 0 dl 1789055758 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 6311.047052] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 144 previous similar messages [ 6316.893636] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.188365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6325.341693] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 11:55:55 (1789055755) [ 6328.884969] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6347.744173] Lustre: lustre-MDT0000-mdc-ffff8c8e852a2000: Connection to lustre-MDT0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6347.756559] Lustre: Skipped 6 previous similar messages [ 6347.759267] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 6347.769895] LustreError: Skipped 4 previous similar messages [ 6347.787394] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a775028527 to 0x4c7c0a775028a21 [ 6347.797877] Lustre: Skipped 4 previous similar messages [ 6347.810720] Lustre: MGC192.168.206.147@tcp: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 6347.819838] Lustre: Skipped 10 previous similar messages [ 6365.668566] LustreError: 2352:0:(client.c:3399:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff8c8e864de680 x1875954387720832/t339302416387(339302416387) o101->lustre-MDT0000-mdc-ffff8c8e852a2000@192.168.206.147@tcp:12/10 lens 592/664 e 0 to 1 dl 1789055796 ref 2 fl Interpret:EXPQU/604/ffffffff rc -110/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 6370.702608] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6372.038444] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6379.166485] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 11:56:49 (1789055809) [ 6380.917139] Lustre: Unmounted lustre-client [ 6420.629853] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 6426.115173] LustreError: lustre-OST0000-osc-ffff8c8e8446b800: operation ost_connect to node 192.168.206.147@tcp failed: rc = -16 [ 6439.308995] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 11:57:49 (1789055869) [ 6440.462925] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs [ 6441.921515] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 11:57:51 (1789055871) [ 6443.295829] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs [ 6444.901318] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 11:57:54 (1789055874) [ 6446.098581] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs [ 6447.585117] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 11:57:57 (1789055877) [ 6448.996903] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs [ 6450.608793] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 11:58:00 (1789055880) [ 6451.918171] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs [ 6453.285681] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 11:58:03 (1789055883) [ 6454.662813] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs [ 6456.015303] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 11:58:06 (1789055886) [ 6457.524175] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs [ 6458.992494] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 11:58:08 (1789055888) [ 6460.446737] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs [ 6462.069533] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 11:58:12 (1789055892) [ 6463.563255] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs [ 6465.268518] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 11:58:15 (1789055895) [ 6466.710930] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs [ 6468.369626] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 11:58:18 (1789055898) [ 6469.611165] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs [ 6471.052803] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 11:58:20 (1789055900) [ 6472.388473] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs [ 6473.818544] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 11:58:23 (1789055903) [ 6475.200654] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs [ 6476.510717] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 11:58:26 (1789055906) [ 6477.676421] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs [ 6478.997282] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 11:58:29 (1789055909) [ 6480.136583] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs [ 6481.428438] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 11:58:31 (1789055911) [ 6482.571467] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs [ 6483.976836] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 11:58:33 (1789055913) [ 6487.088349] LustreError: lustre-MDT0000-mdc-ffff8c8e8446b800: operation mds_statfs to node 192.168.206.147@tcp failed: rc = -107 [ 6487.095700] LustreError: Skipped 1 previous similar message [ 6487.111953] LustreError: lustre-MDT0000-mdc-ffff8c8e8446b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6494.742988] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 11:58:44 (1789055924) [ 6519.264333] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055934/real 1789055934] req@ffff8c8e8950b480 x1875954387867008/t0(0) o400->lustre-MDT0000-mdc-ffff8c8e8446b800@192.168.206.147@tcp:12/10 lens 224/224 e 0 to 1 dl 1789055950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6519.305085] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 6545.757615] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.340680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6555.431427] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 11:59:45 (1789055985) [ 6604.874199] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.504474] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6614.814468] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 12:00:44 (1789056044) [ 6617.027126] Lustre: Unmounted lustre-client [ 6634.297149] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 6642.459266] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 12:01:11 (1789056071) [ 6647.128372] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6681.876683] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6683.505284] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6692.206968] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 12:02:02 (1789056122) [ 6696.235044] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6731.855174] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6733.295501] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6741.915797] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 12:02:51 (1789056171) [ 6746.522215] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6750.142524] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6799.667845] Lustre: 150943:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 6818.814938] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8c8e92059880 x1875954388160768/t356482285584(356482285584) o101->lustre-MDT0000-mdc-ffff8c8e87923800@192.168.206.147@tcp:12/10 lens 576/608 e 0 to 0 dl 1789056265 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 6818.850601] LustreError: 2352:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 101 previous similar messages [ 6851.708133] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 12:04:41 (1789056281) [ 6902.063526] Lustre: Unmounted lustre-client [ 6916.483829] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 6916.530582] LustreError: lustre-OST0000-osc-ffff8c8e894f6800: operation ost_connect to node 192.168.206.147@tcp failed: rc = -16 [ 6952.428076] LustreError: lustre-OST0000-osc-ffff8c8e894f6800: operation ost_connect to node 192.168.206.147@tcp failed: rc = -16 [ 6952.440454] LustreError: Skipped 6 previous similar messages [ 6985.179523] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec [ 6999.544399] Lustre: DEBUG MARKER: free_before: 7520256 free_after: 7520256 [ 7005.410521] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 12:07:15 (1789056435) [ 7013.859422] Lustre: lustre-OST0000-osc-ffff8c8e894f6800: Connection to lustre-OST0000 (at 192.168.206.147@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7013.865304] Lustre: Skipped 8 previous similar messages [ 7032.162267] Lustre: lustre-OST0000-osc-ffff8c8e894f6800: Connection restored to 192.168.206.147@tcp (at 192.168.206.147@tcp) [ 7032.174145] Lustre: Skipped 10 previous similar messages [ 7046.308504] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 12:07:56 (1789056476) [ 7051.747286] LustreError: lustre-OST0000-osc-ffff8c8e894f6800: operation ost_write to node 192.168.206.147@tcp failed: rc = -107 [ 7051.758288] LustreError: Skipped 5 previous similar messages [ 7119.955840] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7121.542594] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7130.360380] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 12:09:19 (1789056559) [ 7150.880187] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789056566/real 1789056566] req@ffff8c8e88728000 x1875954388314112/t0(0) o400->MGC192.168.206.147@tcp@192.168.206.147@tcp:26/25 lens 224/224 e 0 to 1 dl 1789056582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7150.931189] Lustre: 2354:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 7150.941264] LustreError: MGC192.168.206.147@tcp: Connection to MGS (at 192.168.206.147@tcp) was lost; in progress operations using this service will fail [ 7150.964880] LustreError: Skipped 3 previous similar messages [ 7161.324235] Lustre: Evicted from MGS (at 192.168.206.147@tcp) after server handle changed from 0x4c7c0a7750311ba to 0x4c7c0a775031ac7 [ 7161.332638] Lustre: Skipped 3 previous similar messages [ 7162.868703] Lustre: 156546:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.147@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 7265.330793] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7266.739466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7276.935821] Lustre: DEBUG MARKER: == replay-single test complete, duration 7041 sec ======== 12:11:46 (1789056706) [ 7278.606559] Lustre: DEBUG MARKER: === replay-single: start cleanup 12:11:48 (1789056708) === [ 7287.330797] Lustre: DEBUG MARKER: === replay-single: finish cleanup 12:11:56 (1789056716) === [ 7338.213452] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7340.082236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7343.974838] Lustre: Unmounted lustre-client [ 7372.933246] Key type lgssc unregistered [ 7373.219193] LNet: 163689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7373.229805] LNetError: 163689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7373.247419] LNet: Removed LNI 192.168.206.47@tcp [ 7374.035599] Key type .llcrypt unregistered [ 7374.043444] Key type ._llcrypt unregistered