[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 420438222 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003393] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.009553] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012205] LSM: Security Framework initializing [ 0.013062] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014094] *** VALIDATE selinux *** [ 0.026085] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032813] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034181] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036080] *** VALIDATE tmpfs *** [ 0.037527] *** VALIDATE proc *** [ 0.038288] *** VALIDATE cgroup *** [ 0.039012] *** VALIDATE cgroup2 *** [ 0.040360] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041222] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043030] Spectre V2 : User space: Vulnerable [ 0.044012] Speculative Store Bypass: Vulnerable [ 0.048157] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.050316] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051777] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052025] ... version: 2 [ 0.053016] ... bit width: 48 [ 0.054016] ... generic registers: 4 [ 0.055016] ... value mask: 0000ffffffffffff [ 0.056016] ... max period: 00007fffffffffff [ 0.057017] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.059311] rcu: Hierarchical SRCU implementation. [ 0.061376] smp: Bringing up secondary CPUs ... [ 0.063053] x86: Booting SMP configuration: [ 0.064015] .... node #0, CPUs: #1 #2 #3 [ 0.074239] smp: Brought up 1 node, 4 CPUs [ 0.076014] smpboot: Max logical packages: 1 [ 0.077020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132181] node 0 deferred pages initialised in 48ms [ 0.143010] devtmpfs: initialized [ 0.144226] x86/mm: Memory block size: 128MB [ 0.146513] gcov: version magic: 0x41383552 [ 0.150303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160786] pinctrl core: initialized pinctrl subsystem [ 0.163201] [ 0.165009] ************************************************************* [ 0.168016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172012] ** ** [ 0.174012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175011] ** ** [ 0.180014] ** This means that this kernel is built to expose internal ** [ 0.185018] ** IOMMU data structures, which may compromise security on ** [ 0.191014] ** your system. ** [ 0.198012] ** ** [ 0.204013] ** If you see this message and you are not debugging the ** [ 0.209215] ** kernel, report this immediately to your vendor! ** [ 0.218014] ** ** [ 0.227014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232018] ************************************************************* [ 0.236160] NET: Registered protocol family 16 [ 0.238612] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.242104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.246102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.250788] cpuidle: using governor menu [ 0.253107] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.256558] PCI: Using configuration type 1 for base access [ 0.259121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.287455] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.292025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.297216] cryptd: max_cpu_qlen set to 1000 [ 0.300217] ACPI: Added _OSI(Module Device) [ 0.301008] ACPI: Added _OSI(Processor Device) [ 0.302009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.303010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.308000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317534] ACPI: Interpreter enabled [ 0.320070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323013] ACPI: Using IOAPIC for interrupt routing [ 0.325395] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333761] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.345037] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.350019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360297] acpiphp: Slot [2] registered [ 0.361210] acpiphp: Slot [5] registered [ 0.363166] acpiphp: Slot [6] registered [ 0.364113] acpiphp: Slot [3] registered [ 0.367106] acpiphp: Slot [4] registered [ 0.368187] acpiphp: Slot [7] registered [ 0.369000] acpiphp: Slot [8] registered [ 0.372134] acpiphp: Slot [9] registered [ 0.374183] acpiphp: Slot [10] registered [ 0.376125] acpiphp: Slot [11] registered [ 0.378116] acpiphp: Slot [12] registered [ 0.381034] acpiphp: Slot [13] registered [ 0.383122] acpiphp: Slot [14] registered [ 0.385113] acpiphp: Slot [15] registered [ 0.389115] acpiphp: Slot [16] registered [ 0.391185] acpiphp: Slot [17] registered [ 0.392153] acpiphp: Slot [18] registered [ 0.394259] acpiphp: Slot [19] registered [ 0.397140] acpiphp: Slot [20] registered [ 0.398116] acpiphp: Slot [21] registered [ 0.401166] acpiphp: Slot [22] registered [ 0.403143] acpiphp: Slot [23] registered [ 0.405104] acpiphp: Slot [24] registered [ 0.409332] acpiphp: Slot [25] registered [ 0.411067] acpiphp: Slot [26] registered [ 0.412093] acpiphp: Slot [27] registered [ 0.415504] acpiphp: Slot [28] registered [ 0.418230] acpiphp: Slot [29] registered [ 0.420140] acpiphp: Slot [30] registered [ 0.422121] acpiphp: Slot [31] registered [ 0.425077] PCI host bridge to bus 0000:00 [ 0.433174] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.438023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.455024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.460041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.462190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.466932] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.472024] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.486017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.493988] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.499011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.504027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.531015] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.543885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.547044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.554109] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.559389] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.567013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.582013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.589016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.601609] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.612000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.621014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.638031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.662723] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.673016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.683028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.701016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.720620] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.737790] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744627] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.751501] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.765055] iommu: Default domain type: Passthrough [ 0.767840] SCSI subsystem initialized [ 0.771407] ACPI: bus type USB registered [ 0.774116] usbcore: registered new interface driver usbfs [ 0.777127] usbcore: registered new interface driver hub [ 0.781107] usbcore: registered new device driver usb [ 0.785199] pps_core: LinuxPPS API ver. 1 registered [ 0.786000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.796048] PTP clock support registered [ 0.805103] EDAC MC: Ver: 3.0.0 [ 0.806199] PCI: Using ACPI for IRQ routing [ 0.807741] NetLabel: Initializing [ 0.810011] NetLabel: domain hash size = 128 [ 0.815011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820088] NetLabel: unlabeled traffic allowed by default [ 0.824052] vgaarb: loaded [ 0.826256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.831012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.840474] clocksource: Switched to clocksource kvm-clock [ 1.032548] VFS: Disk quotas dquot_6.6.0 [ 1.035344] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.039608] *** VALIDATE ramfs *** [ 1.041574] *** VALIDATE hugetlbfs *** [ 1.044108] pnp: PnP ACPI init [ 1.051295] pnp: PnP ACPI: found 6 devices [ 1.121308] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.133298] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.147322] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.155352] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.167436] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.178549] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.186337] NET: Registered protocol family 2 [ 1.193144] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.204600] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.225905] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.250272] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.262357] TCP: Hash tables configured (established 65536 bind 65536) [ 1.271650] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.285819] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.299379] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.313226] NET: Registered protocol family 1 [ 1.319874] RPC: Registered named UNIX socket transport module. [ 1.326449] RPC: Registered udp transport module. [ 1.330084] RPC: Registered tcp transport module. [ 1.334325] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.338086] NET: Registered protocol family 44 [ 1.340994] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.347527] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.352556] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.360873] PCI: CLS 0 bytes, default 64 [ 1.368520] Unpacking initramfs... [ 2.430404] hrtimer: interrupt took 7569858 ns [ 4.651425] debug: unmapping init [mem 0xffff9881fcc64000-0xffff9881fffcffff] [ 4.656959] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.659820] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.665268] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.208646] Initialise system trusted keyrings [ 6.211184] Key type blacklist registered [ 6.213916] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.237408] zbud: loaded [ 6.243628] *** VALIDATE nfs *** [ 6.246515] *** VALIDATE nfs4 *** [ 6.253216] pstore: using deflate compression [ 6.263662] Platform Keyring initialized [ 6.546101] NET: Registered protocol family 38 [ 6.548376] Key type asymmetric registered [ 6.551093] Asymmetric key parser 'x509' registered [ 6.553101] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.570364] io scheduler mq-deadline registered [ 6.578296] io scheduler kyber registered [ 6.585370] io scheduler bfq registered [ 6.591123] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.596096] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.602191] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.606555] ACPI: Power Button [PWRF] [ 6.615798] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.625403] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.651966] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.706063] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.755341] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.779385] Non-volatile memory driver v1.3 [ 6.785514] Linux agpgart interface v0.103 [ 6.888125] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 6.898322] vda: detected capacity change from 0 to 74690560 [ 6.942143] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.950343] vdb: detected capacity change from 0 to 1073741824 [ 6.963290] libphy: Fixed MDIO Bus: probed [ 6.976755] usbcore: registered new interface driver usbserial_generic [ 6.980887] usbserial: USB Serial support registered for generic [ 6.987136] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.002295] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.006508] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.009667] mousedev: PS/2 mouse device common for all mice [ 7.016070] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.017390] rtc_cmos 00:05: RTC can wake from S4 [ 7.031563] rtc_cmos 00:05: registered as rtc0 [ 7.033678] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.045517] intel_pstate: CPU model not supported [ 7.054182] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.061347] hid: raw HID events driver (C) Jiri Kosina [ 7.076412] usbcore: registered new interface driver usbhid [ 7.083677] usbhid: USB HID core driver [ 7.090666] drop_monitor: Initializing network drop monitor service [ 7.100197] Initializing XFRM netlink socket [ 7.102397] NET: Registered protocol family 10 [ 7.112892] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.124154] Segment Routing with IPv6 [ 7.149690] NET: Registered protocol family 17 [ 7.154772] mpls_gso: MPLS GSO support [ 7.175627] RAS: Correctable Errors collector initialized. [ 7.184704] AVX version of gcm_enc/dec engaged. [ 7.192931] AES CTR mode by8 optimization enabled [ 7.510134] sched_clock: Marking stable (7510099998, 0)->(8890597378, -1380497380) [ 7.527635] registered taskstats version 1 [ 7.529904] Loading compiled-in X.509 certificates [ 7.533346] zswap: loaded using pool lzo/zbud [ 7.627222] Key type big_key registered [ 7.664240] Key type encrypted registered [ 7.666545] ima: No TPM chip found, activating TPM-bypass! [ 7.670552] ima: Allocated hash algorithm: sha1 [ 7.673624] ima: No architecture policies found [ 7.677288] evm: Initialising EVM extended attributes: [ 7.680902] evm: security.selinux [ 7.682964] evm: security.ima [ 7.684936] evm: security.capability [ 7.686906] evm: HMAC attrs: 0x1 [ 7.691853] rtc_cmos 00:05: setting system clock to 2026-09-10 23:01:07 UTC (1789081267) [ 7.705596] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 7.711065] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 7.724104] Write protecting the kernel read-only data: 28672k [ 7.732652] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 7.739405] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 7.827365] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.848338] systemd[1]: Detected virtualization kvm. [ 7.857088] systemd[1]: Detected architecture x86-64. [ 7.861314] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.909487] systemd[1]: No hostname configured. [ 7.914628] systemd[1]: Set hostname to . [ 7.917134] random: systemd: uninitialized urandom read (16 bytes read) [ 7.928432] systemd[1]: Initializing machine ID from random generator. [ 8.058033] random: ln: uninitialized urandom read (6 bytes read) [ 8.306629] random: systemd: uninitialized urandom read (16 bytes read) [ 8.314732] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 8.336004] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 8.390635] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.059661] device-mapper: uevent: version 1.0.3 [ 10.062662] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 11.606333] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.719532] virtio_net virtio0 ens2: renamed from eth0 [ 12.135115] scsi host0: ata_piix [ 12.215296] scsi host1: ata_piix [ 12.220448] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 12.225827] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.204753] random: crng init done [ 17.211419] random: 7 urandom warning(s) missed due to ratelimiting [ 18.127366] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.247755] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.047958] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.770312] SELinux: Disabled at runtime. [ 23.894488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.917783] systemd[1]: Detected virtualization kvm. [ 23.920626] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.583913] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.588587] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.603395] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.615290] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.619345] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.645866] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.657648] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 26.339637] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.569983] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.791511] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.813342] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.339422] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.432665] EDAC sbridge: Ver: 1.1.2 [ 33.281403] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.118483] NFS: Registering the id_resolver key type [ 34.121609] Key type id_resolver registered [ 34.125418] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-client login: [ 106.945036] libcfs: loading out-of-tree module taints kernel. [ 107.045315] Key type ._llcrypt registered [ 107.067311] Key type .llcrypt registered [ 107.941782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 107.967878] alg: No test for adler32 (adler32-zlib) [ 109.668766] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 110.494870] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 112.311349] Key type lgssc registered [ 114.969262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 294.019113] Lustre: Mounted lustre-client [ 299.370669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 319.457990] Lustre: lustre-OST0000-osc-ffff988245490800: disconnect after 23s idle [ 319.778331] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing check_logdir /tmp/testlogs/ [ 325.827083] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing yml_node [ 331.614600] Lustre: DEBUG MARKER: Client: 2.17.55.25 [ 335.108875] Lustre: DEBUG MARKER: MDS: 2.17.55.25 [ 338.045323] Lustre: DEBUG MARKER: OSS: 2.17.55.25 [ 339.655412] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Sep 10 19:06:38 EDT 2026 [ 357.251866] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 359.818869] Lustre: DEBUG MARKER: === replay-single: start setup 19:06:57 (1789081617) === [ 366.887620] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing check_config_client /mnt/lustre [ 389.145450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 404.689492] Lustre: DEBUG MARKER: === replay-single: finish setup 19:07:42 (1789081662) === [ 406.346970] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:07:44 (1789081664) [ 414.680902] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 416.740228] Lustre: lustre-MDT0000-mdc-ffff988245490800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 432.100644] Lustre: lustre-OST0000-osc-ffff988245490800: disconnect after 24s idle [ 432.103508] Lustre: Skipped 1 previous similar message [ 437.218059] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789081681/real 1789081681] req@ffff988248644a80 x1875987786649472/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789081697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 437.259480] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 447.476743] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb060fd21 to 0x4aa145fcb0610150 [ 447.492704] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 450.570729] Lustre: lustre-MDT0000-mdc-ffff988245490800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 462.058393] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 463.744192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 473.369318] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:08:51 (1789081731) [ 478.226276] Lustre: lustre-OST0000-osc-ffff988245490800: Connection to lustre-OST0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 488.418924] Lustre: lustre-OST0001-osc-ffff988245490800: disconnect after 23s idle [ 488.428371] Lustre: Skipped 1 previous similar message [ 494.661577] Lustre: lustre-OST0000-osc-ffff988245490800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 510.935938] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.417491] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 523.657232] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 19:09:41 (1789081781) [ 530.917576] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 531.135528] Lustre: Unmounted lustre-client [ 573.982893] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 579.069970] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 584.193421] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 589.317451] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 594.459258] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 604.670156] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 604.684982] LustreError: Skipped 1 previous similar message [ 625.156376] LustreError: lustre-MDT0000-mdc-ffff988250910000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 625.168284] LustreError: Skipped 3 previous similar messages [ 645.741406] Lustre: Mounted lustre-client [ 656.981591] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 19:11:55 (1789081915) [ 667.083912] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 667.299116] Lustre: Unmounted lustre-client [ 704.317366] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 704.324759] LustreError: Skipped 3 previous similar messages [ 771.079740] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: operation mds_connect to node 192.168.203.133@tcp failed: rc = -16 [ 771.091133] LustreError: Skipped 12 previous similar messages [ 776.214366] Lustre: Mounted lustre-client [ 786.705307] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 19:14:04 (1789082044) [ 794.520843] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 796.658850] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 818.079221] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082061/real 1789082061] req@ffff988248646d80 x1875987786719232/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 818.115301] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 827.378310] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0611117 to 0x4aa145fcb0611593 [ 827.391901] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 839.241691] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 840.854445] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 848.640940] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 19:15:07 (1789082107) [ 856.469891] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 863.211665] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 879.583342] Lustre: 2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082123/real 1789082123] req@ffff988246b26680 x1875987786731776/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 879.615575] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 889.855767] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0611593 to 0x4aa145fcb06119f3 [ 889.886279] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 889.898037] Lustre: Skipped 1 previous similar message [ 894.000176] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988246b25880 x1875987786730240/t21474836484(21474836484) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789082169 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 904.947385] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 906.980229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 915.751827] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 19:16:14 (1789082174) [ 923.281291] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 925.678583] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 942.047608] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082185/real 1789082185] req@ffff988246b24000 x1875987786745472/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 942.082728] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 952.239653] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06119f3 to 0x4aa145fcb0611e30 [ 952.262491] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 952.274527] Lustre: Skipped 1 previous similar message [ 954.021080] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 961.021885] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988246b27b80 x1875987786744576/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789082236 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 969.554745] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 971.187561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 979.760876] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 19:17:17 (1789082237) [ 988.254166] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 992.257428] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1008.607340] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082252/real 1789082252] req@ffff988250879c00 x1875987786759936/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1008.652352] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1018.857313] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0611e30 to 0x4aa145fcb061255a [ 1018.871082] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 1018.883946] Lustre: Skipped 1 previous similar message [ 1022.983493] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff98824503ed80 x1875987786758784/t30064771076(30064771076) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 536/608 e 0 to 0 dl 1789082298 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 1033.388739] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.013611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1043.853578] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 19:18:22 (1789082302) [ 1050.987445] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1054.706129] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1071.071303] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082314/real 1789082314] req@ffff98825087b100 x1875987786771840/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1071.110701] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1080.293781] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb061255a to 0x4aa145fcb061289b [ 1080.304705] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 1080.309806] Lustre: Skipped 1 previous similar message [ 1081.648432] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1098.161833] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.579979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1108.017782] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 19:19:26 (1789082366) [ 1117.984688] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1121.272733] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1124.322114] Lustre: 23309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082368/real 1789082368] req@ffff988246b25c00 x1875987786783232/t0(0) o35->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:23/10 lens 392/624 e 0 to 1 dl 1789082384 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'openfile.0' uid:0 gid:0 projid:0 [ 1137.634095] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1147.889044] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb061289b to 0x4aa145fcb0612d8e [ 1147.911625] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 1147.922816] Lustre: Skipped 1 previous similar message [ 1149.680430] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1163.579950] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1165.112784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1174.173587] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 19:20:32 (1789082432) [ 1182.916957] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1187.815444] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1204.193099] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082447/real 1789082447] req@ffff988248647800 x1875987786800128/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1204.240965] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1204.250025] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1214.461246] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0612d8e to 0x4aa145fcb061338b [ 1226.930217] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1228.375746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1236.561952] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 19:21:35 (1789082495) [ 1243.580486] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1276.903778] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 1276.912395] Lustre: Skipped 3 previous similar messages [ 1288.317535] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1289.807522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1297.609789] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 19:22:36 (1789082556) [ 1305.072489] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1328.993134] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082572/real 1789082572] req@ffff98824503ed80 x1875987786826240/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1329.035015] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1329.045815] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1329.061529] LustreError: Skipped 1 previous similar message [ 1338.346369] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb061389a to 0x4aa145fcb0613d78 [ 1338.366907] Lustre: Skipped 1 previous similar message [ 1352.181549] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.043675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1361.950051] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 19:23:40 (1789082620) [ 1369.521233] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1374.196501] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1374.217680] Lustre: Skipped 2 previous similar messages [ 1405.908976] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988246b24e00 x1875987786838656/t55834574851(55834574851) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789082681 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1417.210217] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.956432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1429.100360] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 19:24:47 (1789082687) [ 1437.198875] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1458.145365] Lustre: 2404:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082701/real 1789082701] req@ffff98824503c000 x1875987786882304/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789082717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1458.202930] Lustre: 2404:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1458.214450] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1458.244019] LustreError: Skipped 1 previous similar message [ 1468.392613] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0614321 to 0x4aa145fcb0614f92 [ 1468.403970] Lustre: Skipped 1 previous similar message [ 1482.420695] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1484.000858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1492.192659] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 19:25:50 (1789082750) [ 1499.324464] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1537.068499] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 1537.078486] Lustre: Skipped 7 previous similar messages [ 1540.135530] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988247f23b80 x1875987786895616/t64424509443(64424509443) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789082815 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1540.200794] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 9 previous similar messages [ 1551.042895] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1552.855427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1578.814059] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 19:27:17 (1789082837) [ 1587.225378] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1633.119866] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.000231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1647.616519] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 19:28:25 (1789082905) [ 1658.186957] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1663.973379] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1664.002933] Lustre: Skipped 3 previous similar messages [ 1704.638142] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1706.427814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1716.109502] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 19:29:34 (1789082974) [ 1724.964653] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1743.839246] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789082987/real 1789082987] req@ffff98824855ad80 x1875987787400704/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789083003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1743.855130] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1743.861600] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 1743.865475] LustreError: Skipped 3 previous similar messages [ 1754.085230] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb061eea1 to 0x4aa145fcb061f3be [ 1754.094568] Lustre: Skipped 3 previous similar messages [ 1756.143749] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 1770.680157] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1772.333699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1780.411418] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 19:30:38 (1789083038) [ 1787.821904] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1825.296724] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988258584700 x1875987787415936/t81604378634(81604378634) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 664/608 e 0 to 0 dl 1789083101 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 1825.327856] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 219 previous similar messages [ 1834.625618] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.163993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1844.672814] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 19:31:43 (1789083103) [ 1851.949526] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1896.980432] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.784406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1906.772489] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 19:32:45 (1789083165) [ 1915.356971] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 1963.424903] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.271391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1974.076917] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 19:33:52 (1789083232) [ 1981.270637] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2018.827957] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988250879f80 x1875987787459584/t94489280522(94489280522) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789083294 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 2026.236520] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2027.734763] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2038.415301] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 19:34:56 (1789083296) [ 2046.175112] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2075.627983] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 2075.633171] Lustre: Skipped 15 previous similar messages [ 2077.504897] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988258587100 x1875987787460352/t94489280524(94489280524) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 576/608 e 0 to 0 dl 1789083353 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 2077.552778] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 1 previous similar message [ 2090.855936] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2092.471298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2100.547493] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 19:35:59 (1789083359) [ 2108.303450] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2152.539332] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.100144] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2162.291641] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 19:37:00 (1789083420) [ 2169.277911] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2175.977190] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2176.003668] Lustre: Skipped 7 previous similar messages [ 2204.527774] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988258587100 x1875987787460352/t94489280524(94489280524) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 576/608 e 0 to 0 dl 1789083480 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 2204.569522] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 3 previous similar messages [ 2216.471140] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2218.239631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2227.320082] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 19:38:05 (1789083485) [ 2235.558796] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2256.607662] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789083500/real 1789083500] req@ffff98825087ad80 x1875987787522432/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789083516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2256.644026] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2256.656555] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 2256.672860] LustreError: Skipped 7 previous similar messages [ 2267.126956] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0621a7d to 0x4aa145fcb06221ca [ 2267.139248] Lustre: Skipped 7 previous similar messages [ 2278.368945] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.775793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2286.868078] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 19:39:05 (1789083545) [ 2293.570565] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2337.482214] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.046399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2347.249534] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 19:40:05 (1789083605) [ 2357.214666] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2386.996266] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988258587100 x1875987787460352/t94489280524(94489280524) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 576/608 e 0 to 0 dl 1789083662 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 2387.032161] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 5 previous similar messages [ 2401.655431] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2403.494686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2411.690681] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 19:41:10 (1789083670) [ 2418.623839] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2459.223524] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2460.438445] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2467.753547] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 19:42:06 (1789083726) [ 2474.016847] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2518.006746] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2519.761491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2527.049883] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:43:05 (1789083785) [ 2533.175232] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2574.487683] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2576.034656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2585.393278] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 19:44:03 (1789083843) [ 2593.268924] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: operation mds_statfs to node 192.168.203.133@tcp failed: rc = -107 [ 2593.288942] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2633.566095] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2635.730239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2649.488419] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2656.981523] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 19:45:15 (1789083915) [ 2660.097092] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: operation mds_statfs to node 192.168.203.133@tcp failed: rc = -107 [ 2660.108270] LustreError: Skipped 1 previous similar message [ 2660.128418] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2668.153514] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:45:26 (1789083926) [ 2675.182319] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2708.465882] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 2708.483189] Lustre: Skipped 21 previous similar messages [ 2710.009444] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988258587800 x1875987787704960/t141733920773(141733920773) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789083985 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 2710.044751] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 8 previous similar messages [ 2716.750389] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.222579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2726.384928] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:46:24 (1789083984) [ 2734.335551] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2766.577122] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2778.813583] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2780.317413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2788.160992] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 19:47:26 (1789084046) [ 2794.859481] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2800.611730] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2800.625908] Lustre: Skipped 11 previous similar messages [ 2818.858440] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2838.025813] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2839.417601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2847.746148] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 19:48:25 (1789084105) [ 2855.088632] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2877.407800] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084121/real 1789084121] req@ffff988250878a80 x1875987787751936/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789084137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2877.441147] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2877.454369] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 2877.461431] LustreError: Skipped 9 previous similar messages [ 2887.658289] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06252df to 0x4aa145fcb0625850 [ 2887.668184] Lustre: Skipped 9 previous similar messages [ 2900.607350] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.253850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2911.120464] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 19:49:29 (1789084169) [ 2918.387477] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 2965.394550] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2967.199989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2976.010267] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:50:34 (1789084234) [ 2983.249891] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3027.766785] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3029.404243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3038.068713] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:51:36 (1789084296) [ 3046.288437] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3091.254864] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3092.891792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3102.036477] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 19:52:40 (1789084360) [ 3109.846326] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3154.180464] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3155.919977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3166.591880] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:53:44 (1789084424) [ 3174.744937] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3222.968620] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3224.433079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3233.553104] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 19:54:51 (1789084491) [ 3242.239962] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3276.908494] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff98824855a680 x1875987787844096/t180388626437(180388626437) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789084552 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 3276.938603] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 15 previous similar messages [ 3290.050832] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3292.286397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3303.125357] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 19:56:00 (1789084560) [ 3311.976351] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3343.349935] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 3343.362982] Lustre: Skipped 19 previous similar messages [ 3357.621924] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3359.487853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3368.584970] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 19:57:06 (1789084626) [ 3370.025586] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3379.815486] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 19:57:17 (1789084637) [ 3386.304889] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3402.256752] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3431.204648] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 19:58:09 (1789084689) [ 3439.357275] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3441.672649] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3441.708359] Lustre: Skipped 10 previous similar messages [ 3457.022566] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3482.878534] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 19:59:00 (1789084740) [ 3491.883716] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3508.209212] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 3508.234908] LustreError: Skipped 9 previous similar messages [ 3508.235423] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3508.268443] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06295cf to 0x4aa145fcb0629dfc [ 3508.283144] Lustre: Skipped 9 previous similar messages [ 3535.972626] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 19:59:54 (1789084794) [ 3553.759192] Lustre: 81049:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084797/real 1789084797] req@ffff988258527b80 x1875987787953152/t0(0) o36->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 512/624 e 0 to 1 dl 1789084813 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3553.810100] Lustre: 81049:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 3554.283273] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3578.229864] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3580.177799] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:00:38 (1789084838) [ 3588.852321] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3610.611666] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3633.446564] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:01:31 (1789084891) [ 3669.918419] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3715.522844] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3717.249517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3740.718417] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:03:19 (1789084999) [ 3768.998943] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 3824.414390] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3826.011883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3847.682651] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:05:06 (1789085106) [ 3857.251726] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:05:15 (1789085115) [ 3883.764520] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 3981.391739] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:07:20 (1789085240) [ 3987.915853] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4021.231612] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 4021.242286] Lustre: Skipped 16 previous similar messages [ 4031.987791] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4033.293254] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4051.536813] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 20:08:30 (1789085310) [ 4056.674742] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4056.694080] Lustre: Skipped 7 previous similar messages [ 4131.354783] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 20:09:50 (1789085390) [ 4153.004327] LustreError: 92327:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4175.851770] LustreError: 92350:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4198.390276] LustreError: 92372:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4221.422735] LustreError: 92394:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4243.661737] LustreError: 92417:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4266.480682] LustreError: 92440:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4289.034797] LustreError: 92463:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4335.093567] LustreError: 92508:0:(lmv_obd.c:1468:lmv_statfs()) lustre-MDT0000-mdc-ffff9882463bb000: can't stat MDS #0: rc = -114 [ 4335.118178] LustreError: 92508:0:(lmv_obd.c:1468:lmv_statfs()) Skipped 1 previous similar message [ 4364.135271] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 20:13:42 (1789085622) [ 4371.687840] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4391.391854] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789085635/real 1789085635] req@ffff988245ce7b80 x1875987790676864/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789085651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4391.403309] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 4391.424399] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4391.438462] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4391.478837] LustreError: Skipped 5 previous similar messages [ 4391.491490] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb066c77d to 0x4aa145fcb066d9a5 [ 4391.523111] Lustre: Skipped 5 previous similar messages [ 4440.177571] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4441.744398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4449.778342] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 20:15:08 (1789085708) [ 4450.066051] Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request [ 4450.089650] LustreError: 95076:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9882463bb000: inode [0x20001a9e1:0x1:0x0] mdc close failed: rc = -108 [ 4450.119541] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4457.016222] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 20:15:15 (1789085715) [ 4491.599169] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4492.944924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4503.655120] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:16:02 (1789085762) [ 4540.157823] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4541.735373] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4614.429355] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:17:52 (1789085872) [ 4622.040566] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4654.065075] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 4654.086045] Lustre: Skipped 20 previous similar messages [ 4659.733230] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988249fb8700 x1875987790754560/t236223201383(236223201383) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789085979 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 [ 4659.769327] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 3 previous similar messages [ 4730.927521] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 20:19:49 (1789085989) [ 4744.575574] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 20:20:03 (1789086003) [ 4752.358663] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4752.367072] Lustre: Skipped 17 previous similar messages [ 4834.102132] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4836.139603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4845.637893] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 20:21:44 (1789086104) [ 4855.469467] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4899.307714] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4900.712980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4909.872825] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 20:22:48 (1789086168) [ 4920.899391] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4965.506217] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4966.780173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4975.345626] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 20:23:53 (1789086233) [ 4983.506809] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 4993.503245] Lustre: 104582:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086237/real 1789086237] req@ffff988245e16d80 x1875987790862976/t0(0) o35->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:23/10 lens 392/624 e 0 to 1 dl 1789086253 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 4993.533652] Lustre: 104582:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 5005.215416] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 5005.231485] LustreError: Skipped 6 previous similar messages [ 5015.534490] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb0671254 to 0x4aa145fcb0671cb1 [ 5015.541161] Lustre: Skipped 6 previous similar messages [ 5023.036382] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 20:24:41 (1789086281) [ 5061.333601] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5062.677553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5070.966571] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 20:25:29 (1789086329) [ 5079.509868] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5123.897849] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5125.618524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5134.283847] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 20:26:32 (1789086392) [ 5144.223807] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5188.506952] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 20:27:26 (1789086446) [ 5199.602266] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5247.964475] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 20:28:26 (1789086506) [ 5259.511829] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5287.400608] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 5287.418824] Lustre: Skipped 17 previous similar messages [ 5288.616129] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 5311.142102] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 20:29:29 (1789086569) [ 5372.895254] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5372.913602] Lustre: Skipped 8 previous similar messages [ 5380.225405] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 20:30:38 (1789086638) [ 5386.835354] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5430.794264] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5432.169788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5450.459856] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 20:31:48 (1789086708) [ 5457.678503] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5488.422646] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 5502.739206] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5504.229459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5522.894451] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 20:33:01 (1789086781) [ 5568.548494] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5610.286420] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5612.087457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5670.328558] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 20:35:28 (1789086928) [ 5671.848139] Lustre: Mounted lustre-client [ 5678.420038] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5697.503193] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789086941/real 1789086941] req@ffff988247db1880 x1875987793722880/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789086957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5697.525598] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 5697.537711] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 5697.547829] LustreError: Skipped 8 previous similar messages [ 5707.755713] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb068a0cf to 0x4aa145fcb06b5210 [ 5707.766671] Lustre: Skipped 8 previous similar messages [ 5709.493017] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 5723.640675] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5725.003472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5729.753132] Lustre: Unmounted lustre-client [ 5735.872763] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 5737.803071] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5744.351492] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 20:36:42 (1789087002) [ 5751.044593] Lustre: Mounted lustre-client [ 5880.470630] Lustre: Unmounted lustre-client [ 5887.894863] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5889.617778] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 20:39:08 (1789087148) [ 5904.990571] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 5938.157895] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 5938.163603] Lustre: Skipped 13 previous similar messages [ 5949.616949] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5950.938485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5960.685802] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 20:40:19 (1789087219) [ 5984.634259] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 6076.763226] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6078.420525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6118.319211] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 20:42:56 (1789087376) [ 6122.469872] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6122.488625] Lustre: Skipped 8 previous similar messages [ 6202.574233] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.997668] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6212.676351] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 20:44:30 (1789087470) [ 6261.595634] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6262.944261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6271.846379] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 20:45:30 (1789087530) [ 6295.147681] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 6317.002764] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 20:46:15 (1789087575) [ 6325.952984] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6350.815931] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789087594/real 1789087594] req@ffff98824497c700 x1875987794638080/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789087610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6350.861610] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 6350.877757] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 6350.887638] LustreError: Skipped 4 previous similar messages [ 6361.065771] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06c9719 to 0x4aa145fcb06ca065 [ 6361.079850] Lustre: Skipped 4 previous similar messages [ 6420.447211] LustreError: 2400:0:(client.c:3398:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff988246bf0380 x1875987794622464/t313532612610(313532612610) o36->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 528/448 e 0 to 1 dl 1789087680 ref 2 fl Interpret:EXQU/204/ffffffff rc -110/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6420.554301] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9882449d9180 x1875987794624768/t313532612612(313532612612) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 592/608 e 0 to 0 dl 1789087740 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 [ 6420.579978] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 19 previous similar messages [ 6427.602906] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6429.222644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6438.824886] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 20:48:17 (1789087697) [ 6500.388714] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 20:49:18 (1789087758) [ 6552.408996] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 20:50:10 (1789087810) [ 6617.904414] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 20:51:16 (1789087876) [ 6644.591491] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6654.687193] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6654.714479] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6664.727129] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6664.773339] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6674.791341] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6674.805122] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6684.807375] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6684.822916] LustreError: 2403:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6694.896472] LustreError: 2403:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6694.913393] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c sleeping for 10000ms [ 6704.929121] LustreError: 132033:0:(client.c:1681:after_reply()) cfs_fail_timeout id 50c awake [ 6712.620902] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 20:52:50 (1789087970) [ 6790.844330] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 20:54:09 (1789088049) [ 6823.870671] Lustre: DEBUG MARKER: phase 2 [ 6834.001738] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 20:54:52 (1789088092) [ 6863.141318] LustreError: 33379:0:(ldlm_request.c:1533:ldlm_cli_cancel_req()) cfs_fail_timeout id 312 sleeping for 19000ms [ 6882.215177] LustreError: 33379:0:(ldlm_request.c:1533:ldlm_cli_cancel_req()) cfs_fail_timeout id 312 awake [ 6915.472336] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 20:56:13 (1789088173) [ 6916.824218] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6918.276682] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 20:56:16 (1789088176) [ 6923.156390] Lustre: DEBUG MARKER: Started rundbench load pid=135362 ... [ 6932.928977] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 6935.856797] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6937.818904] LustreError: lustre-MDT0000-mdc-ffff9882463bb000: operation ldlm_cancel to node 192.168.203.133@tcp failed: rc = -19 [ 6937.824711] Lustre: lustre-MDT0000-mdc-ffff9882463bb000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6937.860699] Lustre: Skipped 4 previous similar messages [ 6961.119480] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789088204/real 1789088204] req@ffff98824497ce00 x1875987795062272/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789088220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6961.156477] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6961.169555] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 6971.377878] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06ca065 to 0x4aa145fcb06d116f [ 6971.392772] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 6971.395934] Lustre: Skipped 9 previous similar messages [ 6971.445796] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff98824497dc00 x1875987794839424/t317827580264(317827580264) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 576/608 e 0 to 0 dl 1789088247 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 [ 6971.463518] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 24 previous similar messages [ 6985.296061] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.812228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6998.080825] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 7000.861716] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 7044.686974] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7046.846174] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7074.587878] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 20:58:53 (1789088333) [ 7204.058262] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 7215.943144] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 7218.268266] LustreError: lustre-MDT0001-mdc-ffff9882463bb000: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19 [ 7218.283110] LustreError: Skipped 1 previous similar message [ 7254.014202] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff988245dbad80 x1875987799663104/t8589938533(8589938533) o35->lustre-MDT0001-mdc-ffff9882463bb000@192.168.203.133@tcp:23/10 lens 392/456 e 0 to 0 dl 1789088524 ref 2 fl Interpret:RQU/604/0 rc -116/-116 job:'tar.0' uid:0 gid:0 projid:0 [ 7254.045557] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 26 previous similar messages [ 7264.048065] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7265.876941] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7325.968503] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 21:03:04 (1789088584) [ 7455.130517] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 7467.461991] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 7470.125888] LustreError: lustre-MDT0001-mdc-ffff9882463bb000: operation ldlm_enqueue to node 192.168.203.133@tcp failed: rc = -107 [ 7494.678757] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988248530700 x1875987801609984/t12884903958(12884903958) o101->lustre-MDT0001-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 648/608 e 0 to 0 dl 1789088770 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 7494.714959] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 62 previous similar messages [ 7509.059197] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7511.137490] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7520.644428] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:06:18 (1789088778) [ 7650.533221] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 7662.065534] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7668.710553] Lustre: lustre-MDT0001-mdc-ffff9882463bb000: Connection to lustre-MDT0001 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7668.721389] Lustre: Skipped 3 previous similar messages [ 7689.266986] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988248530700 x1875987801609984/t12884903958(12884903958) o101->lustre-MDT0001-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 648/608 e 0 to 0 dl 1789088965 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 7689.308257] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 18 previous similar messages [ 7693.835914] Lustre: lustre-MDT0001-mdc-ffff9882463bb000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 7693.849880] Lustre: Skipped 4 previous similar messages [ 7704.839503] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7706.749254] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7717.212132] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:09:35 (1789088975) [ 7729.341661] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 7732.365687] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7740.384171] Lustre: 2403:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789088984/real 1789088984] req@ffff988249c87800 x1875987806463104/t0(0) o4->lustre-OST0000-osc-ffff9882463bb000@192.168.203.133@tcp:6/4 lens 488/448 e 0 to 1 dl 1789089000 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 7775.015461] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7776.605599] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7787.406703] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:10:45 (1789089045) [ 7916.618514] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 7924.742562] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 7936.458162] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7938.834590] LustreError: lustre-MDT0001-mdc-ffff9882463bb000: operation ldlm_cancel to node 192.168.203.133@tcp failed: rc = -19 [ 7960.545118] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 7970.792604] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb06d116f to 0x4aa145fcb07eb8c8 [ 7972.475119] Lustre: 15217:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 7980.089370] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff988249cd5180 x1875987806560256/t322122566255(322122566255) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 520/664 e 0 to 0 dl 1789089255 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 8004.663651] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.736897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.421752] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8017.217317] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:14:35 (1789089275) [ 8025.194276] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8074.207272] LustreError: 2400:0:(client.c:3398:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff988249cd5180 x1875987806560256/t322122566255(322122566255) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 520/664 e 0 to 1 dl 1789089333 ref 2 fl Interpret:EXPQU/604/ffffffff rc -110/-1 job:'lfs.0' uid:0 gid:0 projid:0 [ 8079.629914] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8080.923268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8088.284863] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:15:47 (1789089347) [ 8095.409938] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8140.770541] LustreError: 2400:0:(client.c:3398:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff988249cd5180 x1875987806560256/t322122566255(322122566255) o101->lustre-MDT0000-mdc-ffff9882463bb000@192.168.203.133@tcp:12/10 lens 520/664 e 0 to 1 dl 1789089400 ref 2 fl Interpret:EXPQU/604/ffffffff rc -110/-1 job:'lfs.0' uid:0 gid:0 projid:0 [ 8148.474566] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8150.466875] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8159.021966] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:16:57 (1789089417) [ 8161.621300] Lustre: Unmounted lustre-client [ 8190.919150] Lustre: Mounted lustre-client [ 8196.095344] LustreError: lustre-OST0000-osc-ffff988247ff3800: operation ost_connect to node 192.168.203.133@tcp failed: rc = -16 [ 8196.109986] LustreError: Skipped 1 previous similar message [ 8210.012350] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:17:48 (1789089468) [ 8217.468827] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8261.032910] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8262.505072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8272.036593] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:18:50 (1789089530) [ 8280.063276] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8287.559699] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8289.519306] LustreError: lustre-MDT0001-mdc-ffff988247ff3800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19 [ 8289.538808] LustreError: Skipped 1 previous similar message [ 8289.552061] Lustre: lustre-MDT0001-mdc-ffff988247ff3800: Connection to lustre-MDT0001 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8289.573792] Lustre: Skipped 6 previous similar messages [ 8321.516377] Lustre: lustre-MDT0001-mdc-ffff988247ff3800: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 8321.526851] Lustre: Skipped 9 previous similar messages [ 8328.536276] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8330.125725] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8338.200029] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:19:56 (1789089596) [ 8345.141938] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8351.718452] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8371.167585] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789089614/real 1789089614] req@ffff988249c0b100 x1875987808325504/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789089630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8371.215044] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 8386.122101] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9882441b8380 x1875987808320640/t343597383712(343597383712) o101->lustre-MDT0000-mdc-ffff988247ff3800@192.168.203.133@tcp:12/10 lens 520/664 e 0 to 0 dl 1789089700 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 [ 8386.140455] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 5 previous similar messages [ 8392.869308] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.216775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8432.675989] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.965309] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8442.521609] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:21:40 (1789089700) [ 8453.642754] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8460.220770] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8509.860853] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8511.473721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8513.390250] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8521.939933] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:23:00 (1789089780) [ 8532.159922] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8561.286475] Lustre: 161145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 8575.868983] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8577.476021] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8586.287481] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:24:04 (1789089844) [ 8593.749740] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8630.531399] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8632.157520] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8642.284486] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:25:00 (1789089900) [ 8653.576452] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8662.072556] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8684.960382] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 8684.983201] LustreError: Skipped 6 previous similar messages [ 8694.249609] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb07f6c1a to 0x4aa145fcb07f851f [ 8694.262538] Lustre: Skipped 6 previous similar messages [ 8706.869038] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8708.499879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8747.482720] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.396180] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8759.562759] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:26:57 (1789090017) [ 8770.923749] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8778.082709] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8824.680282] Lustre: 161145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 8845.127592] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8846.784275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8849.022726] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8860.669550] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:28:38 (1789090118) [ 8868.895870] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 8871.132770] LustreError: lustre-MDT0001-mdc-ffff988247ff3800: operation mds_reint to node 192.168.203.133@tcp failed: rc = -19 [ 8911.541297] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.978166] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8921.731724] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:29:40 (1789090180) [ 8929.840688] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8935.912091] Lustre: lustre-MDT0000-mdc-ffff988247ff3800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8935.925797] Lustre: Skipped 14 previous similar messages [ 8961.544528] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 8961.556599] Lustre: Skipped 19 previous similar messages [ 8974.138519] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8975.651995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8983.605534] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:30:41 (1789090241) [ 8991.602858] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 8997.774066] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 9018.850743] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789090262/real 1789090262] req@ffff9882483d5180 x1875987808584576/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789090278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9018.888980] Lustre: 2401:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [ 9040.217891] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9041.874573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9086.166611] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9087.949516] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9096.918400] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 21:32:35 (1789090355) [ 9105.088337] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 9111.760791] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 9160.811381] Lustre: 161145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 9189.557208] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9191.252689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9192.907287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9200.135737] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 21:34:18 (1789090458) [ 9210.603829] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 9242.420184] Lustre: 161145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 9259.534860] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9260.932622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9268.395489] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 21:35:27 (1789090527) [ 9275.787270] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 9309.032577] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9310.497205] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9318.144821] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 21:36:16 (1789090576) [ 9327.363544] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 9337.798294] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 9356.255857] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 9356.287504] LustreError: Skipped 5 previous similar messages [ 9366.504217] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb07fbf41 to 0x4aa145fcb07fca7e [ 9366.515523] Lustre: Skipped 5 previous similar messages [ 9368.114680] Lustre: 161145:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 9384.949413] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9386.532880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9428.481089] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9430.259098] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9438.924214] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 21:38:17 (1789090697) [ 9447.150870] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 9454.870838] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 [ 9507.200411] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9508.772715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9510.577841] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9519.561966] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 21:39:37 (1789090777) [ 9523.419305] LustreError: lustre-MDT0000-mdc-ffff988247ff3800: operation mds_statfs to node 192.168.203.133@tcp failed: rc = -107 [ 9523.428733] LustreError: Skipped 1 previous similar message [ 9523.437716] LustreError: lustre-MDT0000-mdc-ffff988247ff3800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9531.468183] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 21:39:49 (1789090789) [ 9541.609586] Lustre: lustre-MDT0000-mdc-ffff988247ff3800: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9541.632641] Lustre: Skipped 14 previous similar messages [ 9567.717713] Lustre: MGC192.168.203.133@tcp: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [ 9567.728065] Lustre: Skipped 20 previous similar messages [ 9569.013451] LustreError: 2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9882483d4380 x1875987808696960/t390842023944(390842023944) o101->lustre-MDT0000-mdc-ffff988247ff3800@192.168.203.133@tcp:12/10 lens 576/608 e 0 to 0 dl 1789090844 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 [ 9583.311671] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9584.924399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9592.195049] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 21:40:50 (1789090850) [ 9646.526708] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9648.137347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9659.337838] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 21:41:57 (1789090917) [ 9663.189094] Lustre: Unmounted lustre-client [ 9681.635910] Lustre: Mounted lustre-client [ 9688.387835] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 21:42:26 (1789090946) [ 9696.028682] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 9737.267503] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9738.887392] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9747.790432] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 21:43:26 (1789091006) [ 9755.440962] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 9797.795948] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9799.408475] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9807.501398] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 21:44:25 (1789091065) [ 9814.441510] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [ 9821.065619] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 9846.752273] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789091090/real 1789091090] req@ffff988258519500 x1875987809171328/t0(0) o400->MGC192.168.203.133@tcp@192.168.203.133@tcp:26/25 lens 224/224 e 0 to 1 dl 1789091106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9846.777108] Lustre: 2402:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 9858.288630] Lustre: 195848:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 9925.733263] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 21:46:24 (1789091184) [ 9956.327405] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 192.168.203.133@tcp) was lost; in progress operations using this service will fail [ 9956.350460] LustreError: Skipped 3 previous similar messages [ 9966.592276] Lustre: Evicted from MGS (at 192.168.203.133@tcp) after server handle changed from 0x4aa145fcb080445d to 0x4aa145fcb0805925 [ 9966.620321] Lustre: Skipped 3 previous similar messages [ 9974.092105] Lustre: Unmounted lustre-client [ 9990.093148] Lustre: Mounted lustre-client [10036.199912] LustreError: lustre-OST0000-osc-ffff98824a4fe000: operation ost_connect to node 192.168.203.133@tcp failed: rc = -16 [10036.215457] LustreError: Skipped 9 previous similar messages [10054.721387] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 55 sec [10069.270313] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [10077.447782] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 21:48:55 (1789091335) [10122.940600] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 21:49:40 (1789091380) [10189.435975] Lustre: lustre-OST0000-osc-ffff98824a4fe000: Connection restored to 192.168.203.133@tcp (at 192.168.203.133@tcp) [10189.451585] Lustre: Skipped 9 previous similar messages [10195.217481] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10196.959808] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10205.541684] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 21:51:03 (1789091463) [10213.347259] Lustre: lustre-MDT0000-mdc-ffff98824a4fe000: Connection to lustre-MDT0000 (at 192.168.203.133@tcp) was lost; in progress operations using this service will wait for recovery to complete [10213.363140] Lustre: Skipped 9 previous similar messages [10321.389553] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10323.150993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10332.587627] Lustre: DEBUG MARKER: == replay-single test complete, duration 9991 sec ======== 21:53:10 (1789091590) [10334.468838] Lustre: DEBUG MARKER: === replay-single: start cleanup 21:53:12 (1789091592) === [10345.862720] Lustre: DEBUG MARKER: === replay-single: finish cleanup 21:53:24 (1789091604) === [10391.763728] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10393.171271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10397.375884] Lustre: Unmounted lustre-client [10451.319878] Key type lgssc unregistered [10451.750479] LNet: 209194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10451.758921] LNetError: 209194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10451.793181] LNet: Removed LNI 192.168.203.33@tcp [10452.952752] Key type .llcrypt unregistered [10452.956359] Key type ._llcrypt unregistered