[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 418767299 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5410-0x000f541f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5220 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D6F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C0B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BCB (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C7F 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D0F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D47 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c0b-0xbffe1c7e] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c0a] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c7f-0xbffe1d0e] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d0f-0xbffe1d46] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d47-0xbffe1d6e] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003291] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008680] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011168] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.014011] SELinux: Initializing. [ 0.015062] *** VALIDATE selinux *** [ 0.024607] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029696] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033089] *** VALIDATE tmpfs *** [ 0.034483] *** VALIDATE proc *** [ 0.035274] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041022] Spectre V2 : User space: Vulnerable [ 0.042004] Speculative Store Bypass: Vulnerable [ 0.045168] debug: unmapping init [mem 0xffffffffa9e59000-0xffffffffa9e60fff] [ 0.047162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048549] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049017] ... version: 2 [ 0.050007] ... bit width: 48 [ 0.051005] ... generic registers: 4 [ 0.052006] ... value mask: 0000ffffffffffff [ 0.053006] ... max period: 00007fffffffffff [ 0.054006] ... fixed-purpose events: 3 [ 0.055006] ... event mask: 000000070000000f [ 0.056290] rcu: Hierarchical SRCU implementation. [ 0.058332] smp: Bringing up secondary CPUs ... [ 0.059440] x86: Booting SMP configuration: [ 0.060011] .... node #0, CPUs: #1 #2 #3 [ 0.063642] smp: Brought up 1 node, 4 CPUs [ 0.064853] smpboot: Max logical packages: 1 [ 0.065007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.243118] node 0 deferred pages initialised in 176ms [ 0.245115] devtmpfs: initialized [ 0.245735] x86/mm: Memory block size: 128MB [ 0.247036] gcov: version magic: 0x41383552 [ 0.248445] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250152] pinctrl core: initialized pinctrl subsystem [ 0.251088] [ 0.251438] ************************************************************* [ 0.252005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253005] ** ** [ 0.254005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255012] ** ** [ 0.256004] ** This means that this kernel is built to expose internal ** [ 0.257005] ** IOMMU data structures, which may compromise security on ** [ 0.258005] ** your system. ** [ 0.259004] ** ** [ 0.260005] ** If you see this message and you are not debugging the ** [ 0.261005] ** kernel, report this immediately to your vendor! ** [ 0.262005] ** ** [ 0.263005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264005] ************************************************************* [ 0.265464] NET: Registered protocol family 16 [ 0.266318] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267022] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269368] cpuidle: using governor menu [ 0.271514] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272275] PCI: Using configuration type 1 for base access [ 0.274123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.280093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.282092] cryptd: max_cpu_qlen set to 1000 [ 0.283158] ACPI: Added _OSI(Module Device) [ 0.284009] ACPI: Added _OSI(Processor Device) [ 0.284792] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287438] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.291373] ACPI: Interpreter enabled [ 0.292032] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292698] ACPI: Using IOAPIC for interrupt routing [ 0.294060] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.296319] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304224] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306014] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.309030] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.312890] acpiphp: Slot [2] registered [ 0.314047] acpiphp: Slot [5] registered [ 0.314721] acpiphp: Slot [6] registered [ 0.315040] acpiphp: Slot [3] registered [ 0.315761] acpiphp: Slot [4] registered [ 0.317045] acpiphp: Slot [7] registered [ 0.317774] acpiphp: Slot [8] registered [ 0.319040] acpiphp: Slot [9] registered [ 0.319721] acpiphp: Slot [10] registered [ 0.320039] acpiphp: Slot [11] registered [ 0.320732] acpiphp: Slot [12] registered [ 0.321036] acpiphp: Slot [13] registered [ 0.321745] acpiphp: Slot [14] registered [ 0.323038] acpiphp: Slot [15] registered [ 0.323758] acpiphp: Slot [16] registered [ 0.324046] acpiphp: Slot [17] registered [ 0.324863] acpiphp: Slot [18] registered [ 0.326040] acpiphp: Slot [19] registered [ 0.326704] acpiphp: Slot [20] registered [ 0.327053] acpiphp: Slot [21] registered [ 0.327869] acpiphp: Slot [22] registered [ 0.329073] acpiphp: Slot [23] registered [ 0.329859] acpiphp: Slot [24] registered [ 0.330036] acpiphp: Slot [25] registered [ 0.330907] acpiphp: Slot [26] registered [ 0.332056] acpiphp: Slot [27] registered [ 0.332989] acpiphp: Slot [28] registered [ 0.333041] acpiphp: Slot [29] registered [ 0.333941] acpiphp: Slot [30] registered [ 0.335039] acpiphp: Slot [31] registered [ 0.335782] PCI host bridge to bus 0000:00 [ 0.336008] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343008] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.344007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345106] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346652] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348654] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353362] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.355336] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.360532] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362473] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365395] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368007] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.374630] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.377007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380014] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382006] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384661] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392012] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.396710] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.399008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.401846] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.410014] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.417306] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419274] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.421251] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.423209] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.424093] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.427785] iommu: Default domain type: Passthrough [ 0.428251] SCSI subsystem initialized [ 0.429099] ACPI: bus type USB registered [ 0.430038] usbcore: registered new interface driver usbfs [ 0.430936] usbcore: registered new interface driver hub [ 0.431027] usbcore: registered new device driver usb [ 0.431860] pps_core: LinuxPPS API ver. 1 registered [ 0.433005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.435017] PTP clock support registered [ 0.436026] EDAC MC: Ver: 3.0.0 [ 0.436737] PCI: Using ACPI for IRQ routing [ 0.437429] NetLabel: Initializing [ 0.438007] NetLabel: domain hash size = 128 [ 0.438705] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.440048] NetLabel: unlabeled traffic allowed by default [ 0.441049] vgaarb: loaded [ 0.442118] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.443006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.450247] clocksource: Switched to clocksource kvm-clock [ 0.530548] VFS: Disk quotas dquot_6.6.0 [ 0.531716] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.533962] *** VALIDATE ramfs *** [ 0.534772] *** VALIDATE hugetlbfs *** [ 0.536043] pnp: PnP ACPI init [ 0.537675] pnp: PnP ACPI: found 6 devices [ 0.550479] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.551959] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.552953] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.554121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.556631] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.558781] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.560746] NET: Registered protocol family 2 [ 0.562950] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.566761] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.570170] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.575136] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.578152] TCP: Hash tables configured (established 65536 bind 65536) [ 0.580759] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.583360] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.585466] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.587773] NET: Registered protocol family 1 [ 0.590166] RPC: Registered named UNIX socket transport module. [ 0.591902] RPC: Registered udp transport module. [ 0.593218] RPC: Registered tcp transport module. [ 0.594589] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.596450] NET: Registered protocol family 44 [ 0.597959] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.599059] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.600115] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.601247] PCI: CLS 0 bytes, default 64 [ 0.602137] Unpacking initramfs... [ 1.768761] debug: unmapping init [mem 0xffff8a697cc64000-0xffff8a697ffcffff] [ 1.775929] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.778488] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.783267] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.244654] Initialise system trusted keyrings [ 2.245550] Key type blacklist registered [ 2.246776] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.253932] zbud: loaded [ 2.256033] *** VALIDATE nfs *** [ 2.256822] *** VALIDATE nfs4 *** [ 2.258424] pstore: using deflate compression [ 2.262457] Platform Keyring initialized [ 2.368081] NET: Registered protocol family 38 [ 2.369625] Key type asymmetric registered [ 2.370845] Asymmetric key parser 'x509' registered [ 2.372277] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.374837] io scheduler mq-deadline registered [ 2.376392] io scheduler kyber registered [ 2.377868] io scheduler bfq registered [ 2.379499] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.382862] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.385098] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.387560] ACPI: Power Button [PWRF] [ 2.478107] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.567875] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.670814] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.698152] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.725881] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.730974] Non-volatile memory driver v1.3 [ 2.732698] Linux agpgart interface v0.103 [ 2.763272] virtio_blk virtio1: [vda] 133016 512-byte logical blocks (68.1 MB/64.9 MiB) [ 2.765375] vda: detected capacity change from 0 to 68104192 [ 2.781485] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.784133] vdb: detected capacity change from 0 to 1073741824 [ 2.796264] libphy: Fixed MDIO Bus: probed [ 2.802868] usbcore: registered new interface driver usbserial_generic [ 2.804616] usbserial: USB Serial support registered for generic [ 2.806671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.811568] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.814387] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.817082] mousedev: PS/2 mouse device common for all mice [ 2.821139] rtc_cmos 00:05: RTC can wake from S4 [ 2.828147] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.829113] rtc_cmos 00:05: registered as rtc0 [ 2.835711] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.836323] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.840101] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.841118] intel_pstate: CPU model not supported [ 2.860932] hid: raw HID events driver (C) Jiri Kosina [ 2.862697] usbcore: registered new interface driver usbhid [ 2.865311] usbhid: USB HID core driver [ 2.866898] drop_monitor: Initializing network drop monitor service [ 2.869162] Initializing XFRM netlink socket [ 2.871064] NET: Registered protocol family 10 [ 2.873613] Segment Routing with IPv6 [ 2.874938] NET: Registered protocol family 17 [ 2.877083] mpls_gso: MPLS GSO support [ 2.884880] RAS: Correctable Errors collector initialized. [ 2.889085] AVX version of gcm_enc/dec engaged. [ 2.891017] AES CTR mode by8 optimization enabled [ 2.996971] sched_clock: Marking stable (2996802033, 0)->(3755921209, -759119176) [ 3.004715] registered taskstats version 1 [ 3.006526] Loading compiled-in X.509 certificates [ 3.009420] zswap: loaded using pool lzo/zbud [ 3.039694] Key type big_key registered [ 3.055038] Key type encrypted registered [ 3.057124] ima: No TPM chip found, activating TPM-bypass! [ 3.060458] ima: Allocated hash algorithm: sha1 [ 3.061979] ima: No architecture policies found [ 3.064392] evm: Initialising EVM extended attributes: [ 3.067012] evm: security.selinux [ 3.067855] evm: security.ima [ 3.068892] evm: security.capability [ 3.070597] evm: HMAC attrs: 0x1 [ 3.076634] rtc_cmos 00:05: setting system clock to 2025-08-05 14:48:26 UTC (1754405306) [ 3.084141] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.089116] debug: unmapping init [mem 0xffffffffa9b82000-0xffffffffa9e58fff] [ 3.097131] Write protecting the kernel read-only data: 28672k [ 3.102764] debug: unmapping init [mem 0xffffffffa8203000-0xffffffffa83fffff] [ 3.108569] debug: unmapping init [mem 0xffffffffa8b14000-0xffffffffa8bfffff] [ 3.158206] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.166661] systemd[1]: Detected virtualization kvm. [ 3.168654] systemd[1]: Detected architecture x86-64. [ 3.170434] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.205915] systemd[1]: No hostname configured. [ 3.214493] systemd[1]: Set hostname to . [ 3.217810] random: systemd: uninitialized urandom read (16 bytes read) [ 3.220854] systemd[1]: Initializing machine ID from random generator. [ 3.289387] random: ln: uninitialized urandom read (6 bytes read) [ 3.411888] random: systemd: uninitialized urandom read (16 bytes read) [ 3.415800] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.421530] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.426782] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.288174] device-mapper: uevent: version 1.0.3 [ 4.291867] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.607467] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.657844] virtio_net virtio0 ens2: renamed from eth0 [ 6.267803] scsi host0: ata_piix [ 6.283514] scsi host1: ata_piix [ 6.284703] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.286750] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.578491] random: crng init done [ 10.581551] random: 7 urandom warning(s) missed due to ratelimiting [ 12.528079] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.429838] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 16.463167] hrtimer: interrupt took 18011508 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.362335] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.323383] SELinux: Disabled at runtime. [ 19.394577] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.408465] systemd[1]: Detected virtualization kvm. [ 19.411262] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.226191] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.232223] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.236814] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.245749] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.248932] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.261954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.271672] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 21.885258] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.550200] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.330503] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.369558] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.605164] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.681414] EDAC sbridge: Ver: 1.1.2 [ 27.324576] Key type dns_resolver registered [ 28.037883] NFS: Registering the id_resolver key type [ 28.040130] Key type id_resolver registered [ 28.041954] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-client login: [ 74.534848] libcfs: loading out-of-tree module taints kernel. [ 74.563255] Key type ._llcrypt registered [ 74.567109] Key type .llcrypt registered [ 74.897604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 74.904539] alg: No test for adler32 (adler32-zlib) [ 76.004195] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 76.402595] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 78.104144] Key type lgssc registered [ 78.845453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.023351] Lustre: Mounted lustre-client [ 137.622823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 147.611208] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing check_logdir /tmp/testlogs/ [ 149.066276] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing yml_node [ 150.831551] Lustre: DEBUG MARKER: Client: 2.16.57.28 [ 151.926152] Lustre: DEBUG MARKER: MDS: 2.16.57.28 [ 153.017863] Lustre: DEBUG MARKER: OSS: 2.16.57.28 [ 153.704608] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Tue Aug 5 10:50:56 EDT 2025 [ 160.330223] Lustre: DEBUG MARKER: - need MDS1_VERSION <= 2.14.55-100-g8a84c7f9c7 (34617628 <= 34486116) for LU-14927, skip 0f [ 160.736180] Lustre: lustre-OST0000-osc-ffff8a69c525e800: disconnect after 24s idle [ 160.993162] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34617628 < 34486116) for LU-14927, skip 0f [ 161.587799] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 119i 817 411a 130b 130c 130d 130e 130f 130g 312 [ 162.166894] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b [ 162.726201] Lustre: DEBUG MARKER: === sanity: start setup 10:51:05 (1754405465) === [ 164.046745] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing check_config_client /mnt/lustre [ 170.551847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 175.935288] Lustre: DEBUG MARKER: === sanity: finish setup 10:51:18 (1754405478) === [ 181.395940] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 10:51:24 (1754405484) [ 200.160120] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 10:51:42 (1754405502) [ 202.893750] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 10:51:45 (1754405505) [ 205.814144] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 10:51:48 (1754405508) [ 209.008133] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 10:51:51 (1754405511) [ 212.328602] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 10:51:55 (1754405515) [ 215.835152] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 10:51:58 (1754405518) [ 217.056350] Lustre: lustre-OST0000-osc-ffff8a69c525e800: disconnect after 20s idle [ 217.060095] Lustre: Skipped 1 previous similar message [ 219.194400] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 10:52:01 (1754405521) [ 222.113249] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 10:52:04 (1754405524) [ 225.067028] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 10:52:07 (1754405527) [ 230.870896] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 231.708657] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.21787 [ 233.060710] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 233.957906] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.4190 [ 235.173985] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 235.872343] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.7882 [ 237.036455] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 237.769298] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.28305 [ 238.905371] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 239.573834] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.11339 [ 240.875125] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 241.742558] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.3504 [ 243.588474] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 244.473618] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.13531 [ 246.198388] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 246.971782] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.21808 [ 248.287257] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 249.152261] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.13606 [ 250.794059] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 251.547401] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.12282 [ 252.684762] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 253.430767] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.2495 [ 257.937319] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 258.674322] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg141-client.v [ 259.940405] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 260.783536] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg141-client.E [ 262.580698] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 263.636734] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg141-client.v [ 280.032458] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 10:53:01 (1754405581) [ 295.142149] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 10:53:17 (1754405597) [ 302.142731] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 10:53:24 (1754405604) [ 322.039503] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 10:53:43 (1754405623) [ 343.039257] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 10:54:05 (1754405645) [ 352.141405] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 10:54:13 (1754405653) [ 452.815623] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 10:55:55 (1754405755) [ 463.491781] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 10:56:05 (1754405765) [ 474.760261] Lustre: DEBUG MARKER: == sanity test 205k: Verify '?' operator on job stats ==== 10:56:16 (1754405776) [ 482.112161] Lustre: DEBUG MARKER: == sanity test 205l: Verify job stats can scale ========== 10:56:24 (1754405784) [ 575.132905] Lustre: DEBUG MARKER: == sanity test 205m: Test width parsing of job_stats ===== 10:57:57 (1754405877) [ 587.736914] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 10:58:10 (1754405890) [ 587.873486] Lustre: *** cfs_fail_loc=1403, val=1*** [ 587.876144] LustreError: 38578:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000401:0x427:0x0]: rc = -5 [ 587.882358] LustreError: 38578:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5 [ 592.199326] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 10:58:14 (1754405894) [ 596.916848] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 10:58:19 (1754405899) [ 602.121597] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 10:58:24 (1754405904) [ 621.537890] Lustre: lustre-OST0000-osc-ffff8a69c525e800: disconnect after 24s idle [ 621.544352] Lustre: Skipped 1 previous similar message [ 626.656231] Lustre: 2365:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754405914/real 1754405914] req@ffff8a69c9186680 x1839627376606336/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754405930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 626.661596] Lustre: lustre-MDT0000-mdc-ffff8a69c525e800: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 626.661633] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 626.675108] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba70081fa to 0xc2e931ba7030495 [ 626.694579] Lustre: MGC192.168.201.141@tcp: Connection restored to (at 192.168.201.141@tcp) [ 626.717716] LustreError: 2363:0:(mdc_request.c:667:mdc_replay_open()) @@@ cannot properly replay without open data req@ffff8a69c90b2a00 x1839627376591872/t4294971584(4294971584) o101->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 608/608 e 0 to 0 dl 1754405946 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 626.732842] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69c90b2a00 x1839627376591872/t4294971584(4294971584) o101->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 608/608 e 0 to 0 dl 1754405946 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 631.349839] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.776354] Lustre: 2367:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754405919/real 1754405919] req@ffff8a69c8ec1c00 x1839627376606848/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754405935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 631.792202] Lustre: 2367:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 632.390548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 637.024273] Lustre: 2366:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754405924/real 1754405924] req@ffff8a69c90b3100 x1839627376607360/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754405940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.261651] Lustre: lustre-MDT0000-mdc-ffff8a69c525e800: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 652.270497] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 652.283240] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba7030495 to 0xc2e931ba7030800 [ 652.289158] Lustre: MGC192.168.201.141@tcp: Connection restored to (at 192.168.201.141@tcp) [ 652.293104] Lustre: Skipped 1 previous similar message [ 652.298393] LustreError: 2363:0:(mdc_request.c:667:mdc_replay_open()) @@@ cannot properly replay without open data req@ffff8a69c90b2a00 x1839627376591872/t4294971584(4294971584) o101->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 608/608 e 0 to 0 dl 1754405971 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'touch.0' uid:0 gid:0 projid:0 [ 652.315817] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69e0a1d500 x1839627376612352/t8589934595(8589934595) o101->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 584/608 e 0 to 0 dl 1754405971 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 [ 652.331633] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 2 previous similar messages [ 657.610832] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 658.364197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 658.400672] Lustre: 2365:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754405945/real 1754405945] req@ffff8a69c9187100 x1839627376613632/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754405961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.387517] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 10:59:26 (1754405966) [ 664.480111] Lustre: 2366:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754405950/real 1754405950] req@ffff8a69c90b0380 x1839627376614144/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c525e800@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754405966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 668.852509] bash (42291): drop_caches: 3 [ 673.557419] bash (42291): drop_caches: 3 [ 677.579349] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 10:59:40 (1754405980) [ 682.978927] Lustre: lustre-OST0001-osc-ffff8a69c525e800: disconnect after 23s idle [ 682.983877] Lustre: Skipped 1 previous similar message [ 683.054628] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 10:59:45 (1754405985) [ 691.953891] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 10:59:54 (1754405994) [ 692.181427] LustreError: 2366:0:(osc_request.c:3111:osc_enqueue_interpret()) cfs_fail_timeout id 40f sleeping for 10000ms [ 702.192231] LustreError: 2366:0:(osc_request.c:3111:osc_enqueue_interpret()) cfs_fail_timeout id 40f awake [ 706.983325] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 11:00:09 (1754406009) [ 734.948675] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 11:00:37 (1754406037) [ 739.501119] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 11:00:42 (1754406042) [ 750.989901] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 11:00:53 (1754406053) [ 756.146822] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 11:00:58 (1754406058) [ 757.045403] Lustre: DEBUG MARKER: creating a 10 Mb file [ 770.016914] Lustre: lustre-OST0001-osc-ffff8a69c525e800: disconnect after 20s idle [ 791.287113] Lustre: DEBUG MARKER: starting reads [ 792.633922] Lustre: DEBUG MARKER: truncating the file [ 793.953791] Lustre: DEBUG MARKER: killing dd [ 795.024563] Lustre: DEBUG MARKER: removing the temporary file [ 799.160776] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 11:01:41 (1754406101) [ 799.268031] Lustre: *** cfs_fail_loc=411, val=0*** [ 803.880864] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 11:01:46 (1754406106) [ 822.929143] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 11:02:05 (1754406125) [ 835.908155] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 11:02:18 (1754406138) [ 840.839765] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 11:02:23 (1754406143) [ 845.951242] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 11:02:28 (1754406148) [ 850.998488] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 11:02:33 (1754406153) [ 851.161307] Lustre: *** cfs_fail_loc=508, val=2147483648*** [ 851.166984] LustreError: 2356:0:(events.c:193:client_bulk_callback()) event type 1, status -5, desc ffff8a69e07ea000 [ 851.171826] Lustre: 2366:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1754406154/real 1754406154] req@ffff8a69d27f2300 x1839627377268480/t0(0) o4->lustre-OST0000-osc-ffff8a69c525e800@192.168.201.141@tcp:6/4 lens 488/448 e 0 to 1 dl 1754406170 ref 2 fl Rpc:eXQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 851.191368] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 851.215959] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection restored to (at 192.168.201.141@tcp) [ 851.223769] Lustre: Skipped 1 previous similar message [ 856.544922] Lustre: lustre-OST0001-osc-ffff8a69c525e800: disconnect after 20s idle [ 858.319622] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 11:02:40 (1754406160) [ 870.587019] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 11:02:53 (1754406173) [ 883.168175] Lustre: 2366:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406181/real 1754406181] req@ffff8a69d27f3800 x1839627377282432/t0(0) o4->lustre-OST0000-osc-ffff8a69c525e800@192.168.201.141@tcp:6/4 lens 488/448 e 0 to 1 dl 1754406186 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 [ 883.194770] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 883.221929] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection restored to (at 192.168.201.141@tcp) [ 893.207324] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 11:03:15 (1754406195) [ 915.936266] Lustre: 2365:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406199/real 1754406199] req@ffff8a69c8ec2300 x1839627377294208/t0(0) o3->lustre-OST0000-osc-ffff8a69c525e800@192.168.201.141@tcp:6/4 lens 488/440 e 0 to 1 dl 1754406219 ref 2 fl Bulk:RXMQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 [ 915.954653] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 915.961536] LustreError: 2365:0:(client.c:2322:ptlrpc_check_set()) @@@ bulk transfer failed 0/1048576/0 req@ffff8a69c8ec2300 x1839627377294208/t0(0) o3->lustre-OST0000-osc-ffff8a69c525e800@192.168.201.141@tcp:6/4 lens 488/440 e 0 to 1 dl 1754406219 ref 2 fl Bulk:ReXMQU/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 [ 915.970850] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c8ec2300 x1839627377294208/t0(0) o3->lustre-OST0000-osc-ffff8a69c525e800@192.168.201.141@tcp:6/4 lens 488/440 e 0 to 1 dl 1754406219 ref 2 fl Interpret:ReXMQU/600/0 rc -5/0 job:'dd.0' uid:0 gid:0 projid:0 [ 915.985717] Lustre: lustre-OST0000-osc-ffff8a69c525e800: Connection restored to (at 192.168.201.141@tcp) [ 922.225296] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 923.138813] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 924.254818] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 11:03:46 (1754406226) [ 928.898709] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 11:03:51 (1754406231) [ 929.828727] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 931.033946] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 11:03:53 (1754406233) [ 931.995272] Lustre: DEBUG MARKER: SKIP: sanity test_226c needs >= 2 MDTs [ 933.145299] Lustre: DEBUG MARKER: == sanity test 226d: verify fid2path with -n and -fn option ========================================================== 11:03:55 (1754406235) [ 937.646667] Lustre: DEBUG MARKER: == sanity test 226e: Verify path2fid -0 option with newline and space ========================================================== 11:04:00 (1754406240) [ 941.638219] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 11:04:04 (1754406244) [ 945.680222] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 11:04:08 (1754406248) [ 946.730894] Lustre: DEBUG MARKER: SKIP: sanity test_228a ldiskfs only test [ 947.891905] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 11:04:10 (1754406250) [ 949.069108] Lustre: DEBUG MARKER: SKIP: sanity test_228b ldiskfs only test [ 950.245416] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 11:04:12 (1754406252) [ 951.530415] Lustre: DEBUG MARKER: SKIP: sanity test_228c ldiskfs only test [ 952.838630] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 11:04:15 (1754406255) [ 957.111423] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 11:04:19 (1754406259) [ 958.066612] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 959.054905] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 11:04:21 (1754406261) [ 959.991213] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 960.986600] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 11:04:23 (1754406263) [ 961.978586] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 962.925643] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 963.910015] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 11:04:26 (1754406266) [ 964.832945] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 965.921545] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 11:04:28 (1754406268) [ 966.837216] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 967.136591] Lustre: lustre-OST0001-osc-ffff8a69c525e800: disconnect after 21s idle [ 967.827980] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 11:04:30 (1754406270) [ 968.920285] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 970.098937] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 11:04:32 (1754406272) [ 971.116455] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 972.293204] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 11:04:34 (1754406274) [ 973.277978] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 974.308762] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 11:04:36 (1754406276) [ 975.184959] Lustre: DEBUG MARKER: SKIP: sanity test_230j needs >= 2 MDTs [ 976.212201] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 11:04:38 (1754406278) [ 977.084769] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 978.200374] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 11:04:40 (1754406280) [ 979.167276] Lustre: DEBUG MARKER: SKIP: sanity test_230l needs >= 2 MDTs [ 980.145317] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 11:04:42 (1754406282) [ 981.008743] Lustre: DEBUG MARKER: SKIP: sanity test_230m needs >= 2 MDTs [ 982.121044] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 11:04:44 (1754406284) [ 983.005343] Lustre: DEBUG MARKER: SKIP: sanity test_230n needs >= 2 MDTs [ 984.016539] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 11:04:46 (1754406286) [ 984.918803] Lustre: DEBUG MARKER: SKIP: sanity test_230o needs >= 2 MDTs [ 985.927984] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 11:04:48 (1754406288) [ 986.801222] Lustre: DEBUG MARKER: SKIP: sanity test_230p needs >= 2 MDTs [ 987.824316] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 11:04:50 (1754406290) [ 988.795536] Lustre: DEBUG MARKER: SKIP: sanity test_230q needs >= 2 MDTs [ 989.807743] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 11:04:52 (1754406292) [ 990.622427] Lustre: DEBUG MARKER: SKIP: sanity test_230r needs >= 2 MDTs [ 991.528453] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 11:04:54 (1754406294) [ 997.312143] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 11:05:00 (1754406300) [ 998.100640] Lustre: DEBUG MARKER: SKIP: sanity test_230t needs >= 2 MDTs [ 998.990565] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 11:05:01 (1754406301) [ 999.765594] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 1000.627629] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 11:05:03 (1754406303) [ 1001.465556] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 1002.435436] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 11:05:05 (1754406305) [ 1003.190944] Lustre: DEBUG MARKER: SKIP: sanity test_230w needs >= 2 MDTs [ 1004.045376] Lustre: DEBUG MARKER: == sanity test 230x: dir migration check space =========== 11:05:06 (1754406306) [ 1004.836347] Lustre: DEBUG MARKER: SKIP: sanity test_230x needs >= 2 MDTs [ 1005.696667] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 11:05:08 (1754406308) [ 1006.442406] Lustre: DEBUG MARKER: SKIP: sanity test_230y needs >= 2 MDTs [ 1007.278179] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 11:05:10 (1754406310) [ 1008.010776] Lustre: DEBUG MARKER: SKIP: sanity test_230z needs >= 2 MDTs [ 1008.827838] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 11:05:11 (1754406311) [ 1013.698545] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 11:05:16 (1754406316) [ 1037.717953] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 11:05:40 (1754406340) [ 1038.175175] LustreError: lustre-OST0000-osc-ffff8a69c525e800: operation ldlm_enqueue to node 192.168.201.141@tcp failed: rc = -12 [ 1038.893968] LustreError: 71342:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c525e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1038.905917] LustreError: 71342:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1038.933598] Lustre: Unmounted lustre-client [ 1039.240815] Lustre: Mounted lustre-client [ 1044.456963] Lustre: lustre-OST0000-osc-ffff8a69c3eaa800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1045.148966] Lustre: lustre-OST0000-osc-ffff8a69c3eaa800: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 1050.415149] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 11:05:52 (1754406352) [ 1051.979136] LustreError: 72237:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c3eaa800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1051.984151] LustreError: 72237:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 1051.993309] LustreError: 72237:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1051.996698] LustreError: 72237:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1052.023953] Lustre: Unmounted lustre-client [ 1052.245135] Lustre: Mounted lustre-client [ 1063.936615] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 11:06:06 (1754406366) [ 1067.559402] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 11:06:10 (1754406370) [ 1070.990456] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 11:06:13 (1754406373) [ 1071.248501] Lustre: *** cfs_fail_loc=1405, val=0*** [ 1074.991407] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 11:06:17 (1754406377) [ 1080.421420] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 11:06:23 (1754406383) [ 1084.688215] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 11:06:27 (1754406387) [ 1088.300539] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 11:06:31 (1754406391) [ 1142.235557] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 11:07:24 (1754406444) [ 1158.227463] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 11:07:40 (1754406460) [ 1174.433683] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 11:07:57 (1754406477) [ 1175.102910] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 1175.868610] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 11:07:58 (1754406478) [ 1221.679253] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 11:08:44 (1754406524) [ 1244.181537] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 11:09:06 (1754406546) [ 1244.730765] LustreError: lustre-MDT0000-mdc-ffff8a69c9281800: operation mds_readpage to node 192.168.201.141@tcp failed: rc = -12 [ 1247.857039] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 11:09:10 (1754406550) [ 1252.704419] Lustre: 90305:0:(file.c:3144:ll_get_grouplock()) lustre: group lock already exists with gid 97486 on [0x2000013a2:0x1399:0x0]: rc = -22 [ 1252.710152] Lustre: 90305:0:(file.c:3219:ll_put_grouplock()) lustre: no group lock held on [0x2000013a2:0x1399:0x0]: rc = -22 [ 1252.715078] Lustre: 90305:0:(file.c:3126:ll_get_grouplock()) lustre: group id for group lock on [0x2000013a2:0x1399:0x0] is 0: rc = -22 [ 1252.727425] Lustre: 90305:0:(file.c:3229:ll_put_grouplock()) lustre: group lock 4294967286 doesn't match current id 3543 on [0x2000013a2:0x1399:0x0]: rc = -22 [ 1389.731571] Lustre: 90305:0:(file.c:3219:ll_put_grouplock()) lustre: no group lock held on [0x2000013a2:0x13a0:0x0]: rc = -22 [ 1389.748200] Lustre: 90305:0:(file.c:3126:ll_get_grouplock()) lustre: group id for group lock on [0x2000013a2:0x13a0:0x0] is 0: rc = -22 [ 1392.827359] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 11:11:35 (1754406695) [ 1433.380270] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 11:12:16 (1754406736) [ 1436.528493] Lustre: DEBUG MARKER: == sanity test 245a: check mdc connection flag/data: multiple modify RPCs ========================================================== 11:12:19 (1754406739) [ 1439.263803] Lustre: DEBUG MARKER: == sanity test 245b: check osp connection flag/data: multiple modify RPCs ========================================================== 11:12:22 (1754406742) [ 1440.075381] Lustre: DEBUG MARKER: SKIP: sanity test_245b needs >= 2 MDTs [ 1440.838485] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 11:12:23 (1754406743) [ 1441.026495] Lustre: Mounted lustre-client [ 1441.102057] LustreError: 93071:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c2bf0000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1441.107126] LustreError: 93071:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 1441.112715] LustreError: 93071:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1441.114764] LustreError: 93071:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1441.131172] Lustre: Unmounted lustre-client [ 1444.351916] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 11:12:27 (1754406747) [ 1444.535622] LustreError: 93706:0:(llite_lib.c:501:client_common_fill_super()) lustre-clilmv-ffff8a69d8f27000: cannot mds_connect: rc = -2 [ 1444.550480] LustreError: 93706:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1444.553346] LustreError: 93706:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1444.560880] Lustre: Unmounted lustre-client [ 1444.562484] Lustre: Skipped 1 previous similar message [ 1444.564553] LustreError: 93706:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1447.129421] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 11:12:29 (1754406749) [ 1447.317380] Lustre: Mounted lustre-client [ 1447.318650] Lustre: Skipped 1 previous similar message [ 1447.394626] LustreError: 94327:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69e0d2a000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1447.398881] LustreError: 94327:0:(lov_obd.c:784:lov_cleanup()) Skipped 3 previous similar messages [ 1450.485677] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 11:12:33 (1754406753) [ 1450.776552] LustreError: 94988:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1450.779498] LustreError: 94988:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1450.795172] Lustre: Unmounted lustre-client [ 1450.796891] Lustre: Skipped 2 previous similar messages [ 1454.029837] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 11:12:36 (1754406756) [ 1454.242673] Lustre: Mounted lustre-client [ 1454.244329] Lustre: Skipped 3 previous similar messages [ 1454.324062] LustreError: 95655:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69da3f6000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1454.328634] LustreError: 95655:0:(lov_obd.c:784:lov_cleanup()) Skipped 7 previous similar messages [ 1454.454152] LustreError: lustre-MDT0000-mdc-ffff8a69c2bf5000: operation mds_get_root to node 192.168.201.141@tcp failed: rc = -22 [ 1454.458121] LustreError: 95662:0:(llite_lib.c:501:client_common_fill_super()) lustre-clilmv-ffff8a69c2bf5000: cannot mds_connect: rc = -22 [ 1454.474096] LustreError: 95662:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 1457.436085] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 11:12:40 (1754406760) [ 1458.204589] Lustre: DEBUG MARKER: SKIP: sanity test_247f needs >= 2 MDTs [ 1459.049777] Lustre: DEBUG MARKER: == sanity test 247g: striped directory submount revalidate ROOT from cache ========================================================== 11:12:41 (1754406761) [ 1459.731932] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs > 1 MDTs [ 1460.487073] Lustre: DEBUG MARKER: == sanity test 247h: remote directory submount revalidate ROOT from cache ========================================================== 11:12:43 (1754406763) [ 1461.130706] Lustre: DEBUG MARKER: SKIP: sanity test_247h needs > 1 MDTs [ 1461.875762] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 11:12:44 (1754406764) [ 1529.953734] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 11:13:52 (1754406832) [ 1559.267892] Lustre: DEBUG MARKER: == sanity test 248c: verify whole file read behavior ===== 11:14:22 (1754406862) [ 1570.666696] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 11:14:33 (1754406873) [ 1573.175159] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 11:14:36 (1754406876) [ 1573.791413] Lustre: DEBUG MARKER: SKIP: sanity test_250 no 16TB file size limit on ZFS [ 1574.458715] Lustre: DEBUG MARKER: == sanity test 251a: Handling short read and write correctly ========================================================== 11:14:37 (1754406877) [ 1575.084051] Lustre: *** cfs_fail_loc=1407, val=0*** [ 1577.572670] Lustre: DEBUG MARKER: == sanity test 251b: short read restore offset correctly ========================================================== 11:14:40 (1754406880) [ 1577.643457] LustreError: 100387:0:(file.c:2465:do_file_read_iter()) cfs_fail_timeout id 1431 sleeping for 5000ms [ 1582.744140] LustreError: 100387:0:(file.c:2465:do_file_read_iter()) cfs_fail_timeout id 1431 awake [ 1585.156892] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 11:14:47 (1754406887) [ 1585.974474] Lustre: DEBUG MARKER: SKIP: sanity test_252 ldiskfs only test [ 1586.693166] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 11:14:49 (1754406889) [ 1587.411716] Lustre: DEBUG MARKER: SKIP: sanity test_253 need >= 2.13.57 and ldiskfs for fallocate [ 1588.094451] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 11:14:50 (1754406890) [ 1593.935971] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 1594.568277] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 1595.217046] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 1595.862544] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 1596.528133] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 11:14:59 (1754406899) [ 1597.129342] LustreError: lustre-MDT0000-mdc-ffff8a69c9281800: operation ldlm_enqueue to node 192.168.201.141@tcp failed: rc = -14 [ 1603.044796] Lustre: lustre-MDT0000-mdc-ffff8a69c9281800: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1603.045215] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 1603.050112] Lustre: Skipped 1 previous similar message [ 1603.057880] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba7045b17 to 0xc2e931ba7110074 [ 1603.063391] Lustre: MGC192.168.201.141@tcp: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 1603.065870] Lustre: Skipped 1 previous similar message [ 1603.074888] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69d27f1f80 x1839627378022528/t12884905701(12884905701) o101->lustre-MDT0000-mdc-ffff8a69c9281800@192.168.201.141@tcp:12/10 lens 648/608 e 0 to 0 dl 1754406922 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 [ 1603.084330] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 2 previous similar messages [ 1606.428120] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 11:15:09 (1754406909) [ 1608.817875] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 11:15:11 (1754406911) [ 1611.300735] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 11:15:14 (1754406914) [ 1611.848764] Lustre: DEBUG MARKER: SKIP: sanity test_259 ldiskfs only test [ 1612.524050] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 11:15:15 (1754406915) [ 1612.595155] Lustre: *** cfs_fail_loc=806, val=0*** [ 1612.596862] Lustre: 105230:0:(mdc_request.c:918:mdc_close()) lustre-MDT0000-mdc-ffff8a69c9281800: close of FID [0x2000013a2:0x13d3:0x0] failed, file reference will be dropped when this client unmounts or is evicted [ 1612.602782] LustreError: 105230:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8a69c9281800: inode [0x2000013a2:0x13d3:0x0] mdc close failed: rc = -12 [ 1615.127347] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 11:15:17 (1754406917) [ 1620.224719] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 11:15:23 (1754406923) [ 1622.955675] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 11:15:25 (1754406925) [ 1625.879171] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 11:15:28 (1754406928) [ 1628.618907] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 11:15:31 (1754406931) [ 1632.053553] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 11:15:34 (1754406934) [ 1638.737716] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 11:15:41 (1754406941) [ 1648.657895] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 11:15:51 (1754406951) [ 1652.655482] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 11:15:55 (1754406955) [ 1655.574413] Lustre: DEBUG MARKER: == sanity test 270j: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 11:15:58 (1754406958) [ 1658.821109] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 11:16:01 (1754406961) [ 1661.794794] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 11:16:04 (1754406964) [ 1664.687468] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 11:16:07 (1754406967) [ 1667.681650] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 11:16:10 (1754406970) [ 1706.736389] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 11:16:49 (1754407009) [ 1710.452392] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 11:16:53 (1754407013) [ 1713.481337] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 11:16:56 (1754407016) [ 1714.595378] Lustre: *** cfs_fail_loc=314, val=0*** [ 1717.072146] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 11:16:59 (1754407019) [ 1720.138659] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 11:17:02 (1754407022) [ 1724.598363] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 11:17:07 (1754407027) [ 1728.500636] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 11:17:11 (1754407031) [ 1729.690961] LustreError: 118686:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8a69c9281800: inode [0x2000013a2:0x1c0b:0x0] mdc close failed: rc = -22 [ 1732.400458] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 11:17:15 (1754407035) [ 1734.089203] LustreError: 119280:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8a69c9281800: inode [0x2000013a2:0x1c0f:0x0] mdc close failed: rc = -22 [ 1736.917349] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 11:17:19 (1754407039) [ 1737.784426] LustreError: 119874:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8a69c9281800: inode [0x2000013a2:0x1c14:0x0] mdc close failed: rc = -22 [ 1740.895024] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 11:17:23 (1754407043) [ 1744.202308] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 11:17:27 (1754407047) [ 1749.961585] Lustre: DEBUG MARKER: == sanity test 273c: race writeback and object destroy === 11:17:32 (1754407052) [ 1755.708386] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 11:17:38 (1754407058) [ 1756.222729] LustreError: 75413:0:(ldlm_lockd.c:2916:ldlm_bl_thread_blwi()) cfs_fail_timeout id 31f sleeping for 4000ms [ 1758.304065] LustreError: 75413:0:(ldlm_lockd.c:2916:ldlm_bl_thread_blwi()) cfs_fail_timeout interrupted [ 1760.402107] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 11:17:43 (1754407063) [ 1761.763404] Lustre: lustre-OST0000-osc-ffff8a69c9281800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1773.722713] Lustre: lustre-OST0000-osc-ffff8a69c9281800: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 1773.727311] Lustre: Skipped 1 previous similar message [ 1828.323330] Lustre: lustre-OST0000-osc-ffff8a69c9281800: Connection to lustre-OST0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1828.327336] Lustre: Skipped 9 previous similar messages [ 1838.101142] Lustre: lustre-OST0000-osc-ffff8a69c9281800: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 1838.105139] Lustre: Skipped 9 previous similar messages [ 1894.915084] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 11:19:57 (1754407197) [ 1897.389655] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 11:20:00 (1754407200) [ 1897.926757] Lustre: DEBUG MARKER: SKIP: sanity test_278 needs >= 2 MDTs [ 1898.486922] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 11:20:01 (1754407201) [ 1898.865980] LustreError: 128889:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c9281800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1898.868984] LustreError: 128889:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 1898.873564] LustreError: 128889:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1898.875331] LustreError: 128889:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 1898.889708] Lustre: Unmounted lustre-client [ 1898.891554] Lustre: Skipped 3 previous similar messages [ 1904.610180] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 1904.615541] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1904.617229] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba714055a to 0xc2e931ba7140599 [ 1904.632275] LustreError: 128911:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -5 [ 1920.334881] Lustre: Mounted lustre-client [ 1922.786217] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 11:20:25 (1754407225) [ 1923.349824] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 1923.991926] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 11:20:26 (1754407226) [ 1924.508730] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 1925.089351] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 1925.710561] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 1926.386345] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 11:20:29 (1754407229) [ 1927.023845] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 1927.727839] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ========================================================== 11:20:30 (1754407230) [ 1928.345924] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 1929.042884] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ========================================================== 11:20:31 (1754407231) [ 1929.644231] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 1930.243288] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ========================================================== 11:20:33 (1754407233) [ 1930.778208] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 1931.367689] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory ========================================================== 11:20:34 (1754407234) [ 1931.930701] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 1932.577202] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory ========================================================== 11:20:35 (1754407235) [ 1933.165328] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 1933.786178] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ============ 11:20:36 (1754407236) [ 1934.332337] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 1934.936719] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ======== 11:20:37 (1754407237) [ 1935.457460] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 1936.108351] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ========================================================== 11:20:38 (1754407238) [ 1936.666713] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 1937.251162] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS ========================================================== 11:20:40 (1754407240) [ 1939.758484] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA ========================================================== 11:20:42 (1754407242) [ 1940.276348] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 1940.823737] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 1941.390742] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================================== 11:20:44 (1754407244) [ 1941.894959] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 1942.496899] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ========================================================== 11:20:45 (1754407245) [ 1943.049210] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 1943.682726] Lustre: DEBUG MARKER: == sanity test 300r: test -1 striped directory =========== 11:20:46 (1754407246) [ 1944.209780] Lustre: DEBUG MARKER: SKIP: sanity test_300r needs >= 2 MDTs [ 1944.760206] Lustre: DEBUG MARKER: == sanity test 300s: test lfs mkdir -c without -i ======== 11:20:47 (1754407247) [ 1945.316485] Lustre: DEBUG MARKER: SKIP: sanity test_300s needs >= 2 MDTs [ 1945.966892] Lustre: DEBUG MARKER: == sanity test 300t: test max_mdt_stripecount ============ 11:20:48 (1754407248) [ 1946.507195] Lustre: DEBUG MARKER: SKIP: sanity test_300t needs at least 2 MDTs [ 1947.987163] Lustre: DEBUG MARKER: == sanity test 300ua: basic overstriped dir sanity test == 11:20:50 (1754407250) [ 1948.656322] Lustre: DEBUG MARKER: SKIP: sanity test_300ua needs >= 2 MDTs [ 1949.339228] Lustre: DEBUG MARKER: == sanity test 300ub: test MDT overstriping interface [ 1949.887067] Lustre: DEBUG MARKER: SKIP: sanity test_300ub needs >= 2 MDTs [ 1950.529514] Lustre: DEBUG MARKER: == sanity test 300uc: test MDT overstriping as default [ 1951.108397] Lustre: DEBUG MARKER: SKIP: sanity test_300uc needs >= 2 MDTs [ 1951.810224] Lustre: DEBUG MARKER: == sanity test 300ud: dir split ========================== 11:20:54 (1754407254) [ 1952.432455] Lustre: DEBUG MARKER: SKIP: sanity test_300ud needs >= 2 MDTs [ 1953.147974] Lustre: DEBUG MARKER: == sanity test 300ue: dir merge ========================== 11:20:55 (1754407255) [ 1953.822635] Lustre: DEBUG MARKER: SKIP: sanity test_300ue needs >= 2 MDTs [ 1954.631450] Lustre: DEBUG MARKER: == sanity test 300uf: migrate with too many local locks == 11:20:57 (1754407257) [ 1955.299567] Lustre: DEBUG MARKER: SKIP: sanity test_300uf needs >= 2 MDTs [ 1956.064635] Lustre: DEBUG MARKER: == sanity test 300ug: migrate overstriped dirs =========== 11:20:58 (1754407258) [ 1956.861514] Lustre: DEBUG MARKER: SKIP: sanity test_300ug needs >= 2 MDTs [ 1957.762740] Lustre: DEBUG MARKER: == sanity test 300uh: overstripe tunable max_stripes_per_mdt ========================================================== 11:21:00 (1754407260) [ 1958.492337] Lustre: DEBUG MARKER: SKIP: sanity test_300uh needs >= 2 MDTs [ 1959.247199] Lustre: DEBUG MARKER: == sanity test 300ui: overstripe is not supported on one MDT system ========================================================== 11:21:02 (1754407262) [ 1961.929743] Lustre: DEBUG MARKER: == sanity test 300uj: overstriped dir with -C -N sanity test ========================================================== 11:21:04 (1754407264) [ 1962.525355] Lustre: DEBUG MARKER: SKIP: sanity test_300uj needs >= 2 MDTs [ 1963.708773] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ============= 11:21:06 (1754407266) [ 1964.350728] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 1965.121894] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open ========================================================== 11:21:07 (1754407267) [ 1965.775655] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 1966.532455] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ========================================================== 11:21:09 (1754407269) [ 1967.147446] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 1967.866535] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ========================================================== 11:21:10 (1754407270) [ 2020.339519] Lustre: DEBUG MARKER: SKIP: sanity test_312 skipping ALWAYS excluded test 312 [ 2020.995779] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ========================================================== 11:22:03 (1754407323) [ 2024.449614] Lustre: DEBUG MARKER: == sanity test 314: OSP shouldn't fail after last_rcvd update failure ========================================================== 11:22:07 (1754407327) [ 2045.514659] Lustre: DEBUG MARKER: == sanity test 315: read should be accounted ============= 11:22:28 (1754407348) [ 2052.213478] Lustre: DEBUG MARKER: == sanity test 316: lfs migrate of file with large_xattr enabled ========================================================== 11:22:35 (1754407355) [ 2052.855853] Lustre: DEBUG MARKER: SKIP: sanity test_316 needs >= 2 MDTs [ 2053.588714] Lustre: DEBUG MARKER: == sanity test 317: Verify blocks get correctly update after truncate ========================================================== 11:22:36 (1754407356) [ 2054.194026] Lustre: DEBUG MARKER: SKIP: sanity test_317 LU-10370: no implementation for ZFS [ 2054.957505] Lustre: DEBUG MARKER: == sanity test 318: Verify async readahead tunables ====== 11:22:37 (1754407357) [ 2055.092690] LustreError: 144333:0:(lproc_llite.c:1814:read_ahead_async_file_threshold_mb_store()) lustre: can't set read_ahead_async_file_threshold_mb=65 > max_read_readahead_per_file_mb=64 [ 2057.919460] Lustre: DEBUG MARKER: == sanity test 319: lost lease lock on migrate error ===== 11:22:40 (1754407360) [ 2058.559543] Lustre: DEBUG MARKER: SKIP: sanity test_319 needs >= 2 MDTs [ 2059.272791] Lustre: DEBUG MARKER: == sanity test 350: force NID mismatch path to be exercised ========================================================== 11:22:42 (1754407362) [ 2109.772109] Lustre: DEBUG MARKER: == sanity test 360: ldiskfs unlink in a separate thread == 11:23:32 (1754407412) [ 2110.485276] Lustre: DEBUG MARKER: SKIP: sanity test_360 ldiskfs only test [ 2111.234964] Lustre: DEBUG MARKER: == sanity test 398a: direct IO should cancel lock otherwise lockless ========================================================== 11:23:34 (1754407414) [ 2114.332631] Lustre: DEBUG MARKER: == sanity test 398b: DIO and buffer IO race ============== 11:23:37 (1754407417) [ 2276.094596] Lustre: DEBUG MARKER: == sanity test 398c: run fio to test AIO ================= 11:26:18 (1754407578) [ 2364.139147] Lustre: DEBUG MARKER: == sanity test 398d: run aiocp to verify block size > stripe size ========================================================== 11:27:46 (1754407666) [ 2382.072877] Lustre: DEBUG MARKER: == sanity test 398e: O_Direct open cleared by fcntl doesn't cause hang ========================================================== 11:28:04 (1754407684) [ 2384.707952] Lustre: DEBUG MARKER: == sanity test 398f: verify aio handles ll_direct_rw_pages errors correctly ========================================================== 11:28:07 (1754407687) [ 2391.902281] Lustre: DEBUG MARKER: == sanity test 398g: verify parallel dio async RPC submission ========================================================== 11:28:14 (1754407694) [ 2415.178400] Lustre: DEBUG MARKER: == sanity test 398h: verify correctness of read [ 2430.924368] Lustre: DEBUG MARKER: == sanity test 398i: verify parallel dio handles ll_direct_rw_pages errors correctly ========================================================== 11:28:53 (1754407733) [ 2433.307338] Lustre: *** cfs_fail_loc=1418, val=0*** [ 2435.928222] Lustre: DEBUG MARKER: == sanity test 398j: test parallel dio where stripe size > rpc_size ========================================================== 11:28:58 (1754407738) [ 2451.103467] Lustre: DEBUG MARKER: == sanity test 398k: test enospc on first stripe ========= 11:29:13 (1754407753) [ 2470.555494] Lustre: DEBUG MARKER: SKIP: sanity test_398k 7528448 > 600000 skipping out-of-space test on OST0 [ 2471.204975] Lustre: DEBUG MARKER: == sanity test 398l: test enospc on intermediate stripe/RPC ========================================================== 11:29:34 (1754407774) [ 2481.074551] Lustre: DEBUG MARKER: SKIP: sanity test_398l 7507968 > 600000 skipping out-of-space test on OST0 [ 2499.248235] Lustre: DEBUG MARKER: == sanity test 398m: test RPC failures with parallel dio ========================================================== 11:30:02 (1754407802) [ 2499.706819] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2499.710263] LustreError: 2367:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69d8407b80 x1839627413504896/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407819 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'dd.0' uid:0 gid:0 projid:0 [ 2500.835348] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2500.835483] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69d8407b80 x1839627413505792/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407820 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:0 [ 2500.839371] LustreError: Skipped 4 previous similar messages [ 2500.847507] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 6 previous similar messages [ 2502.882995] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2502.883224] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c7d06a00 x1839627413507712/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407822 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2502.885470] LustreError: Skipped 3 previous similar messages [ 2502.893758] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 3 previous similar messages [ 2505.954647] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2505.957830] LustreError: Skipped 2 previous similar messages [ 2505.959553] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c8cd8380 x1839627413508224/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407825 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:0 [ 2509.986936] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2509.990797] LustreError: Skipped 3 previous similar messages [ 2509.992224] LustreError: 2367:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69d8407b80 x1839627413509120/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407829 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_00.0' uid:0 gid:0 projid:0 [ 2509.998202] LustreError: 2367:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 3 previous similar messages [ 2521.573034] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2521.573388] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c8ce5500 x1839627413511168/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407840 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2521.578906] LustreError: Skipped 8 previous similar messages [ 2521.588200] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 7 previous similar messages [ 2546.147065] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2546.147189] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c8cdb100 x1839627413515136/t0(0) o4->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407865 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_01_01.0' uid:0 gid:0 projid:0 [ 2546.150188] LustreError: Skipped 11 previous similar messages [ 2546.155441] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 10 previous similar messages [ 2556.388305] LustreError: 2367:0:(osc_request.c:2605:brw_interpret()) lustre-OST0000-osc-ffff8a69c9231800: too many resent retries for object: 9663677440:4211: rc = -5 [ 2556.392485] LustreError: 2367:0:(osc_request.c:2605:brw_interpret()) Skipped 1 previous similar message [ 2580.964374] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_read to node 192.168.201.141@tcp failed: rc = -5 [ 2580.967269] LustreError: Skipped 30 previous similar messages [ 2580.968514] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c8ce6680 x1839627413544192/t0(0) o3->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/440 e 0 to 0 dl 1754407900 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2580.974350] LustreError: 2365:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 26 previous similar messages [ 2615.779383] LustreError: 2365:0:(osc_request.c:2605:brw_interpret()) lustre-OST0000-osc-ffff8a69c9231800: too many resent retries for object: 9663677440:4211: rc = -5 [ 2615.783501] LustreError: 2365:0:(osc_request.c:2605:brw_interpret()) Skipped 2 previous similar messages [ 2645.475139] LustreError: lustre-OST0001-osc-ffff8a69c9231800: operation ost_write to node 192.168.201.141@tcp failed: rc = -5 [ 2645.477391] LustreError: Skipped 47 previous similar messages [ 2645.478422] LustreError: 2364:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c7d04a80 x1839627413562880/t0(0) o4->lustre-OST0001-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/224 e 0 to 0 dl 1754407964 ref 2 fl Interpret:ReMQU/600/0 rc -5/-5 job:'ptlrpcd_00_00.0' uid:0 gid:0 projid:0 [ 2645.483334] LustreError: 2364:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 43 previous similar messages [ 2672.035800] LustreError: 2365:0:(osc_request.c:2605:brw_interpret()) lustre-OST0001-osc-ffff8a69c9231800: too many resent retries for object: 10737419264:3107: rc = -5 [ 2672.040107] LustreError: 2365:0:(osc_request.c:2605:brw_interpret()) Skipped 3 previous similar messages [ 2730.467719] LustreError: 2366:0:(osc_request.c:2605:brw_interpret()) lustre-OST0001-osc-ffff8a69c9231800: too many resent retries for object: 10737419264:3106: rc = -5 [ 2730.471044] LustreError: 2366:0:(osc_request.c:2605:brw_interpret()) Skipped 3 previous similar messages [ 2733.900810] Lustre: DEBUG MARKER: == sanity test 398n: test append with parallel DIO ======= 11:33:56 (1754408036) [ 2745.725092] Lustre: DEBUG MARKER: == sanity test 398o: right kms with DIO ================== 11:34:08 (1754408048) [ 2748.313166] Lustre: DEBUG MARKER: == sanity test 398p: race aio with buffered i/o ========== 11:34:11 (1754408051) [ 2824.037454] Lustre: DEBUG MARKER: == sanity test 398q: race dio with buffered i/o ========== 11:35:26 (1754408126) [ 2976.113719] Lustre: DEBUG MARKER: == sanity test 398r: i/o error on file read ============== 11:37:58 (1754408278) [ 2976.759396] LustreError: lustre-OST0000-osc-ffff8a69c9231800: operation ost_read to node 192.168.201.141@tcp failed: rc = -5 [ 2976.765293] LustreError: Skipped 59 previous similar messages [ 2976.767705] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff8a69c92dca80 x1839627418990720/t0(0) o3->lustre-OST0000-osc-ffff8a69c9231800@192.168.201.141@tcp:6/4 lens 488/4536 e 0 to 0 dl 1754408296 ref 2 fl Interpret:RMQU/600/0 rc -5/-5 job:'cat.0' uid:0 gid:0 projid:0 [ 2976.779320] LustreError: 2366:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 51 previous similar messages [ 3032.549753] LustreError: 2366:0:(osc_request.c:2605:brw_interpret()) lustre-OST0000-osc-ffff8a69c9231800: too many resent retries for object: 9663677440:4223: rc = -5 [ 3032.554450] LustreError: 2366:0:(osc_request.c:2605:brw_interpret()) Skipped 3 previous similar messages [ 3035.981749] Lustre: DEBUG MARKER: == sanity test 398s: i/o error on mirror file read ======= 11:38:58 (1754408338) [ 3040.435856] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write ========================================================== 11:39:03 (1754408343) [ 3137.176193] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ========================================================== 11:40:39 (1754408439) [ 3138.188939] Lustre: DEBUG MARKER: SKIP: sanity test_399b ldiskfs only test [ 3139.056486] Lustre: DEBUG MARKER: SKIP: sanity test_400a skipping excluded test 400a [ 3139.913933] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ==== 11:40:42 (1754408442) [ 3143.555230] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ========================================================== 11:40:46 (1754408446) [ 3147.346593] Lustre: DEBUG MARKER: == sanity test 401aa: Verify that 'lctl list_param -p' lists the correct path names ========================================================== 11:40:49 (1754408449) [ 3150.999875] Lustre: DEBUG MARKER: == sanity test 401ab: Check that 'lctl list_param -r' lists only readable params ========================================================== 11:40:53 (1754408453) [ 3154.415601] Lustre: DEBUG MARKER: == sanity test 401ac: Check that 'lctl list_param -w' lists only writable params ========================================================== 11:40:57 (1754408457) [ 3158.222627] Lustre: DEBUG MARKER: == sanity test 401ad: Check that 'lctl list_param -wr' is conjunctive ========================================================== 11:41:00 (1754408460) [ 3161.736549] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ========================================================== 11:41:04 (1754408464) [ 3165.001419] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. ========================================================== 11:41:07 (1754408467) [ 3168.390526] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' ========================================================== 11:41:11 (1754408471) [ 3172.094834] Lustre: DEBUG MARKER: == sanity test 401db: Verify 'lctl set_param' does not add trailing '=' ========================================================== 11:41:14 (1754408474) [ 3271.558922] Lustre: DEBUG MARKER: == sanity test 401e: verify 'lctl get_param' works with NID in parameter ========================================================== 11:42:54 (1754408574) [ 3274.407190] Lustre: DEBUG MARKER: == sanity test 401f: check 'lctl list_param' doesn't follow symlinks with --no-links ========================================================== 11:42:57 (1754408577) [ 3277.283504] Lustre: DEBUG MARKER: == sanity test 401ga: check 'set_param -C' sets params upon mount ========================================================== 11:43:00 (1754408580) [ 3277.789685] LustreError: 167231:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c9231800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3277.794783] LustreError: 167231:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 3277.798191] LustreError: 167231:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 3277.800647] LustreError: 167231:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3277.817957] Lustre: Unmounted lustre-client [ 3277.819247] Lustre: Skipped 1 previous similar message [ 3277.990471] Lustre: Mounted lustre-client [ 3280.973320] Lustre: DEBUG MARKER: == sanity test 401gb: check 'set_param -d -C' removes client params ========================================================== 11:43:03 (1754408583) [ 3281.401172] LustreError: 167888:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c525f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3281.405968] LustreError: 167888:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 3281.412198] LustreError: 167888:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 3281.415275] LustreError: 167888:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3281.432458] Lustre: Unmounted lustre-client [ 3281.583591] Lustre: Mounted lustre-client [ 3284.426930] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ========================================================== 11:43:07 (1754408587) [ 3287.651677] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ========================================================== 11:43:10 (1754408590) [ 3288.012622] sysctl (169135): drop_caches: 2 [ 3290.902723] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ========================================================== 11:43:13 (1754408593) [ 3296.678279] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests ======= 11:43:19 (1754408599) [ 3299.577354] Lustre: DEBUG MARKER: SKIP: sanity test_405 layout swap does not support DOM files so far [ 3300.525378] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ====== 11:43:23 (1754408603) [ 3301.358643] Lustre: DEBUG MARKER: SKIP: sanity test_406 needs >= 2 MDTs [ 3302.164702] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 3303.018287] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ========================================================== 11:43:25 (1754408605) [ 3303.173486] Lustre: *** cfs_fail_loc=40a, val=0*** [ 3303.175439] LustreError: 2364:0:(osc_request.c:2872:osc_build_rpc()) lustre-OST0001-osc-ffff8a69c4e19000: prep_req failed: rc = -22 [ 3303.179731] LustreError: 2364:0:(osc_cache.c:2217:osc_check_rpcs()) Read request failed with -22 [ 3307.976695] bash (170977): drop_caches: 2 [ 3311.130705] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file ========================================================== 11:43:33 (1754408613) [ 3311.829377] Lustre: DEBUG MARKER: SKIP: sanity test_409 needs >= 2 MDTs [ 3312.675071] Lustre: DEBUG MARKER: == sanity test 410: Test inode number returned from kernel thread ========================================================== 11:43:35 (1754408615) [ 3312.844969] lustre_kinode_15346: CONFIG_X86_X32 is not set [ 3312.852867] lustre_kinode_15346: inode is 144115339507007594 [ 3312.856161] lustre_kinode_15346: inode is 144115339507007594 [ 3312.857871] lustre_kinode_15346: inode numbers are identical: 144115339507007594 [ 3315.806690] Lustre: DEBUG MARKER: SKIP: sanity test_411a skipping ALWAYS excluded test 411a [ 3316.611683] Lustre: DEBUG MARKER: == sanity test 411b: confirm Lustre can avoid OOM with reasonable cgroups limits ========================================================== 11:43:39 (1754408619) [ 3762.638537] Lustre: DEBUG MARKER: SKIP: sanity test_411b OST space are too small: 3762176K [ 3763.500810] Lustre: DEBUG MARKER: == sanity test 412: mkdir on specific MDTs =============== 11:51:06 (1754409066) [ 3764.078259] Lustre: DEBUG MARKER: SKIP: sanity test_412 needs >= 2 MDTs [ 3764.818070] Lustre: DEBUG MARKER: == sanity test 413a: QoS mkdir with 'lfs mkdir -i -1' ==== 11:51:07 (1754409067) [ 3765.462802] Lustre: DEBUG MARKER: SKIP: sanity test_413a We need at least 2 MDTs for this test [ 3766.194641] Lustre: DEBUG MARKER: == sanity test 413b: QoS mkdir under dir whose default LMV starting MDT offset is -1 ========================================================== 11:51:09 (1754409069) [ 3766.836369] Lustre: DEBUG MARKER: SKIP: sanity test_413b We need at least 2 MDTs for this test [ 3767.561964] Lustre: DEBUG MARKER: == sanity test 413c: mkdir with default LMV max inherit rr ========================================================== 11:51:10 (1754409070) [ 3768.221048] Lustre: DEBUG MARKER: SKIP: sanity test_413c We need at least 2 MDTs for this test [ 3768.956812] Lustre: DEBUG MARKER: == sanity test 413d: inherit ROOT default LMV ============ 11:51:11 (1754409071) [ 3769.634621] Lustre: DEBUG MARKER: SKIP: sanity test_413d We need at least 2 MDTs for this test [ 3770.449114] Lustre: DEBUG MARKER: == sanity test 413e: check default max-inherit value ===== 11:51:13 (1754409073) [ 3771.170773] Lustre: DEBUG MARKER: SKIP: sanity test_413e We need at least 2 MDTs for this test [ 3771.984559] Lustre: DEBUG MARKER: == sanity test 413f: lfs getdirstripe -D list ROOT default LMV if it's not set on dir ========================================================== 11:51:14 (1754409074) [ 3772.634854] Lustre: DEBUG MARKER: SKIP: sanity test_413f We need at least 2 MDTs for this test [ 3773.387197] Lustre: DEBUG MARKER: == sanity test 413g: enforce ROOT default LMV on subdir mount ========================================================== 11:51:16 (1754409076) [ 3774.049344] Lustre: DEBUG MARKER: SKIP: sanity test_413g We need at least 2 MDTs for this test [ 3774.809579] Lustre: DEBUG MARKER: == sanity test 413h: don't stick to parent for round-robin dirs ========================================================== 11:51:17 (1754409077) [ 3775.444671] Lustre: DEBUG MARKER: SKIP: sanity test_413h We need at least 2 MDTs for this test [ 3776.189388] Lustre: DEBUG MARKER: == sanity test 413i: check default layout inheritance ==== 11:51:19 (1754409079) [ 3776.842209] Lustre: DEBUG MARKER: SKIP: sanity test_413i needs >= 2 MDTs [ 3777.657777] Lustre: DEBUG MARKER: == sanity test 413j: set default LMV by setxattr ========= 11:51:20 (1754409080) [ 3778.322409] Lustre: DEBUG MARKER: SKIP: sanity test_413j needs >= 2 MDTs [ 3779.049355] Lustre: DEBUG MARKER: == sanity test 413k: QoS mkdir exclude prefixes ========== 11:51:21 (1754409081) [ 3782.344279] Lustre: DEBUG MARKER: == sanity test 413z: 413 test cleanup ==================== 11:51:25 (1754409085) [ 3784.904252] Lustre: DEBUG MARKER: == sanity test 414: simulate ENOMEM in ptlrpc_register_bulk() ========================================================== 11:51:27 (1754409087) [ 3785.477709] LustreError: 177518:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c4e19000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3785.482786] LustreError: 177518:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 3785.489381] LustreError: 177518:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 3785.491888] LustreError: 177518:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3785.519148] Lustre: Unmounted lustre-client [ 3785.710240] Lustre: Mounted lustre-client [ 3788.200713] Lustre: DEBUG MARKER: == sanity test 415: lock revoke is not missing =========== 11:51:31 (1754409091) [ 3807.396985] Lustre: DEBUG MARKER: == sanity test 416: transaction start failure won't cause system hung ========================================================== 11:51:50 (1754409110) [ 3810.121592] Lustre: DEBUG MARKER: == sanity test 417: disable remote dir, striped dir and dir migration ========================================================== 11:51:53 (1754409113) [ 3810.668531] Lustre: DEBUG MARKER: SKIP: sanity test_417 needs >= 2 MDTs [ 3811.302855] Lustre: DEBUG MARKER: == sanity test 418: df and lfs df outputs match ========== 11:51:54 (1754409114) [ 3859.411630] Lustre: DEBUG MARKER: == sanity test 419: Verify open file by name doesn't crash kernel ========================================================== 11:52:42 (1754409162) [ 3862.533364] Lustre: DEBUG MARKER: == sanity test 420: clear SGID bit on non-directories for non-members ========================================================== 11:52:45 (1754409165) [ 3865.367915] Lustre: DEBUG MARKER: == sanity test 421a: simple rm by fid ==================== 11:52:48 (1754409168) [ 3867.889058] Lustre: DEBUG MARKER: == sanity test 421b: rm by fid on open file ============== 11:52:50 (1754409170) [ 3870.423386] Lustre: DEBUG MARKER: == sanity test 421c: rm by fid against hardlinked files == 11:52:53 (1754409173) [ 3874.848238] Lustre: DEBUG MARKER: == sanity test 421d: rmfid en masse ====================== 11:52:57 (1754409177) [ 3909.225877] Lustre: DEBUG MARKER: == sanity test 421e: rmfid in DNE ======================== 11:53:32 (1754409212) [ 3909.756369] Lustre: DEBUG MARKER: SKIP: sanity test_421e needs >= 2 MDTs [ 3910.566144] Lustre: DEBUG MARKER: == sanity test 421f: rmfid checks permissions ============ 11:53:33 (1754409213) [ 3911.046248] Lustre: Mounted lustre-client [ 3913.188957] LustreError: 186199:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c3eae000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3913.192941] LustreError: 186199:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 3913.197649] LustreError: 186199:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3913.199660] LustreError: 186199:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3913.213122] Lustre: Unmounted lustre-client [ 3913.764275] Lustre: DEBUG MARKER: == sanity test 421g: rmfid to return errors properly ===== 11:53:36 (1754409216) [ 3914.297141] Lustre: DEBUG MARKER: SKIP: sanity test_421g needs >= 2 MDTs [ 3914.879561] Lustre: DEBUG MARKER: == sanity test 421h: rmfid with fileset mount ============ 11:53:37 (1754409217) [ 3918.015360] Lustre: DEBUG MARKER: == sanity test 422: kill a process with RPC in progress == 11:53:40 (1754409220) [ 3939.808107] Lustre: 187460:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754409222/real 1754409222] req@ffff8a69f5c4ed80 x1839627427980288/t0(0) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/1376 e 0 to 1 dl 1754409242 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3939.816824] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3939.821994] Lustre: Skipped 9 previous similar messages [ 3939.829342] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 3939.831710] Lustre: Skipped 9 previous similar messages [ 3960.288196] Lustre: 187460:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754409243/real 1754409243] req@ffff8a69f5c4ed80 x1839627427980288/t0(0) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/1376 e 0 to 1 dl 1754409263 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3960.288322] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3960.297029] Lustre: 187460:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3960.310756] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 3962.641523] Lustre: DEBUG MARKER: touch [ 3980.768112] Lustre: 187460:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754409263/real 1754409263] req@ffff8a69f5c4ed80 x1839627427980288/t0(0) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/1376 e 0 to 1 dl 1754409283 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 [ 3984.236860] Lustre: DEBUG MARKER: == sanity test 423: statfs should return a right data ==== 11:54:47 (1754409287) [ 3988.589855] Lustre: DEBUG MARKER: == sanity test 424: simulate ENOMEM in ptl_send_rpc bulk reply ME attach ========================================================== 11:54:51 (1754409291) [ 3988.709170] Lustre: *** cfs_fail_loc=522, val=0*** [ 3988.710289] LustreError: 2365:0:(niobuf.c:1040:ptl_send_rpc()) LNetMEAttach failed: -12 [ 3991.059934] Lustre: DEBUG MARKER: == sanity test 425: lock count should not exceed lru size ========================================================== 11:54:53 (1754409293) [ 4001.294247] Lustre: DEBUG MARKER: == sanity test 426: splice test on Lustre ================ 11:55:04 (1754409304) [ 4004.258325] Lustre: DEBUG MARKER: == sanity test 427: Failed DNE2 update request shouldn't corrupt updatelog ========================================================== 11:55:07 (1754409307) [ 4005.005317] Lustre: DEBUG MARKER: SKIP: sanity test_427 needs >= 2 MDTs [ 4005.679121] Lustre: DEBUG MARKER: == sanity test 428: large block size IO should not hang == 11:55:08 (1754409308) [ 4085.111954] Lustre: DEBUG MARKER: == sanity test 429: verify if opencache flag on client side does work ========================================================== 11:56:27 (1754409387) [ 4087.501654] Lustre: DEBUG MARKER: == sanity test 430a: lseek: SEEK_DATA/SEEK_HOLE basic functionality ========================================================== 11:56:30 (1754409390) [ 4088.049666] Lustre: DEBUG MARKER: SKIP: sanity test_430a MDT does not support SEEK_HOLE [ 4088.642667] Lustre: DEBUG MARKER: == sanity test 430b: lseek: SEEK_DATA/SEEK_HOLE special cases ========================================================== 11:56:31 (1754409391) [ 4089.211584] Lustre: DEBUG MARKER: SKIP: sanity test_430b OST does not support SEEK_HOLE [ 4089.854124] Lustre: DEBUG MARKER: == sanity test 430c: lseek: external tools check ========= 11:56:32 (1754409392) [ 4090.474991] Lustre: DEBUG MARKER: SKIP: sanity test_430c OST does not support SEEK_HOLE [ 4091.111206] Lustre: DEBUG MARKER: == sanity test 431: Restart transaction for IO =========== 11:56:33 (1754409393) [ 4093.856260] bash (192813): drop_caches: 3 [ 4096.630233] Lustre: DEBUG MARKER: == sanity test 432: mv dir from outside Lustre =========== 11:56:39 (1754409399) [ 4144.777692] Lustre: DEBUG MARKER: == sanity test 433: ldlm lock cancel releases dentries and inodes ========================================================== 11:57:27 (1754409447) [ 4157.039397] Lustre: DEBUG MARKER: == sanity test 434: Client should not send RPCs for security.selinux with SElinux disabled ========================================================== 11:57:39 (1754409459) [ 4162.348286] Lustre: DEBUG MARKER: == sanity test 440: bash completion for lfs, lctl ======== 11:57:45 (1754409465) [ 4164.703128] Lustre: DEBUG MARKER: == sanity test 442: truncate vs read/write should not panic ========================================================== 11:57:47 (1754409467) [ 4165.807568] LustreError: 196927:0:(llite_lib.c:3166:ll_truncate_inode_pages_final()) cfs_fail_timeout id 1430 sleeping for 5000ms [ 4170.904125] LustreError: 196927:0:(llite_lib.c:3166:ll_truncate_inode_pages_final()) cfs_fail_timeout id 1430 awake [ 4173.368705] Lustre: DEBUG MARKER: == sanity test 460d: Check encrypt pools output ========== 11:57:56 (1754409476) [ 4176.169335] Lustre: DEBUG MARKER: == sanity test 600a: basic test for mlock()ed file ======= 11:57:59 (1754409479) [ 4176.858216] Lustre: DEBUG MARKER: SKIP: sanity test_600a This test needs vmtouch utility [ 4177.455079] Lustre: DEBUG MARKER: == sanity test 600b: mlock a file (via vmtouch) larger than max_cached_mb ========================================================== 11:58:00 (1754409480) [ 4178.164906] Lustre: DEBUG MARKER: SKIP: sanity test_600b This test needs vmtouch utility [ 4178.814523] Lustre: DEBUG MARKER: == sanity test 600c: Test I/O when mlocked page count > @max_cached_mb ========================================================== 11:58:01 (1754409481) [ 4179.557033] Lustre: DEBUG MARKER: SKIP: sanity test_600c This test needs vmtouch utility [ 4180.220889] Lustre: DEBUG MARKER: == sanity test 600d: Test I/O with limited LRU page slots (some was mlocked) ========================================================== 11:58:03 (1754409483) [ 4180.950673] Lustre: DEBUG MARKER: SKIP: sanity test_600d This test needs vmtouch utility [ 4181.563707] Lustre: DEBUG MARKER: == sanity test 801a: write barrier user interfaces and stat machine ========================================================== 11:58:04 (1754409484) [ 4225.450744] Lustre: DEBUG MARKER: == sanity test 801b: modification will be blocked by write barrier ========================================================== 11:58:47 (1754409527) [ 4231.984636] Lustre: 200633:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69c612ed80 x1839627432047232/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754409590 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 4250.931204] Lustre: DEBUG MARKER: == sanity test 801c: rescan barrier bitmap =============== 11:59:12 (1754409552) [ 4252.781371] Lustre: DEBUG MARKER: SKIP: sanity test_801c needs >= 2 MDTs [ 4254.801243] Lustre: DEBUG MARKER: == sanity test 802b: be able to set MDTs to readonly ===== 11:59:16 (1754409556) [ 4264.264813] Lustre: DEBUG MARKER: == sanity test 802c: be able to set OFDs to readonly ===== 11:59:26 (1754409566) [ 4270.668522] Lustre: DEBUG MARKER: == sanity test 803a: verify agent object for remote object ========================================================== 11:59:33 (1754409573) [ 4271.613573] Lustre: DEBUG MARKER: SKIP: sanity test_803a needs >= 2 MDTs [ 4272.529398] Lustre: DEBUG MARKER: == sanity test 803b: remote object can getattr from cache ========================================================== 11:59:35 (1754409575) [ 4273.310969] Lustre: DEBUG MARKER: SKIP: sanity test_803b needs >= 2 MDTs [ 4274.222938] Lustre: DEBUG MARKER: == sanity test 804: verify agent entry for remote entry == 11:59:36 (1754409576) [ 4275.029357] Lustre: DEBUG MARKER: SKIP: sanity test_804 needs >= 2 MDTs [ 4275.869948] Lustre: DEBUG MARKER: == sanity test 805: ZFS can remove from full fs ========== 11:59:38 (1754409578) [ 4363.725225] LustreError: 204320:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8a69c7c6b000: inode [0x200002343:0x2222:0x0] mdc close failed: rc = -122 [ 4415.248594] Lustre: DEBUG MARKER: == sanity test 806: Verify Lazy Size on MDS ============== 12:01:57 (1754409717) [ 4428.472584] Lustre: DEBUG MARKER: == sanity test 807a: verify LSOM syncing tool ============ 12:02:11 (1754409731) [ 4431.668039] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing cancel_lru_locks osc [ 4440.892270] Lustre: DEBUG MARKER: == sanity test 807b: verify lfs somsync utility ========== 12:02:23 (1754409743) [ 4442.477840] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing cancel_lru_locks osc [ 4450.487848] Lustre: DEBUG MARKER: == sanity test 808: Check trusted.som xattr not logged in Changelogs ========================================================== 12:02:33 (1754409753) [ 4455.661727] Lustre: DEBUG MARKER: == sanity test 809: Verify no SOM xattr store for DoM-only files ========================================================== 12:02:38 (1754409758) [ 4458.281775] Lustre: DEBUG MARKER: == sanity test 810: partial page writes on ZFS (LU-11663) ========================================================== 12:02:41 (1754409761) [ 4458.396967] Lustre: *** cfs_fail_loc=411, val=0*** [ 4458.398518] Lustre: Skipped 1 previous similar message [ 4458.988969] Lustre: *** cfs_fail_loc=411, val=0*** [ 4458.990502] Lustre: Skipped 9 previous similar messages [ 4460.055819] Lustre: *** cfs_fail_loc=411, val=0*** [ 4460.058340] Lustre: Skipped 17 previous similar messages [ 4463.897484] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 4464.517101] Lustre: DEBUG MARKER: == sanity test 812a: do not drop reqs generated when imp is going to idle (LU-11951) ========================================================== 12:02:47 (1754409767) [ 4465.811671] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4466.276601] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in FULL state after 0 sec [ 4467.682493] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4478.387299] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in CONNECTING state after 10 sec [ 4481.264342] Lustre: DEBUG MARKER: == sanity test 812b: do not drop no resend request for idle connect ========================================================== 12:03:04 (1754409784) [ 4482.462229] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4482.937939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in FULL state after 0 sec [ 4484.487472] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state CONNECTING osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4494.254137] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in CONNECTING state after 9 sec [ 4495.954191] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4508.740230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in IDLE state after 12 sec [ 4511.371292] Lustre: DEBUG MARKER: == sanity test 812c: idle import vs lock enqueue race ==== 12:03:34 (1754409814) [ 4512.700872] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4513.171823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in FULL state after 0 sec [ 4523.488405] LustreError: 155610:0:(import.c:2064:ptlrpc_disconnect_and_idle_import()) cfs_race id 533 sleeping [ 4528.241666] LustreError: 213336:0:(osc_lock.c:1027:osc_lock_enqueue()) cfs_fail_race id 533 waking [ 4528.244246] LustreError: 155610:0:(import.c:2064:ptlrpc_disconnect_and_idle_import()) cfs_fail_race id 533 awake: rc=247 [ 4528.755479] LustreError: 213336:0:(osc_lock.c:1027:osc_lock_enqueue()) cfs_fail_race id 533 waking [ 4531.979632] Lustre: DEBUG MARKER: == sanity test 813: File heat verfication ================ 12:03:54 (1754409834) [ 4662.549524] Lustre: DEBUG MARKER: == sanity test 814: sparse cp works as expected (LU-12361) ========================================================== 12:06:05 (1754409965) [ 4665.158668] Lustre: DEBUG MARKER: == sanity test 815: zero byte tiny write doesn't hang (LU-12382) ========================================================== 12:06:08 (1754409968) [ 4667.616269] Lustre: DEBUG MARKER: == sanity test 816: do not reset lru_resize on idle reconnect ========================================================== 12:06:10 (1754409970) [ 4668.873481] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4669.353597] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in FULL state after 0 sec [ 4670.590579] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid 50 [ 4683.370560] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a69c7c6b000.ost_server_uuid in IDLE state after 12 sec [ 4685.867795] Lustre: DEBUG MARKER: SKIP: sanity test_817 skipping ALWAYS excluded test 817 [ 4686.457178] Lustre: DEBUG MARKER: == sanity test 818: unlink with failed llog ============== 12:06:29 (1754409989) [ 4687.330779] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4687.335355] Lustre: Skipped 1 previous similar message [ 4692.451877] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 4692.460198] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba71e4002 to 0xc2e931ba72dba2c [ 4692.465728] Lustre: MGC192.168.201.141@tcp: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 4692.469501] Lustre: Skipped 1 previous similar message [ 4692.505263] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69c7895880 x1839627434580608/t21474882100(21474882100) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/608 e 0 to 0 dl 1754410011 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 4692.514182] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 7 previous similar messages [ 4711.954121] LustreError: lustre-MDT0000-mdc-ffff8a69c7c6b000: operation ldlm_enqueue to node 192.168.201.141@tcp failed: rc = -107 [ 4711.956299] LustreError: Skipped 19 previous similar messages [ 4711.957572] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4711.992581] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69c7895880 x1839627434580608/t21474882100(21474882100) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/608 e 0 to 0 dl 1754410031 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 4711.997828] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 4 previous similar messages [ 4712.022419] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 4712.025668] Lustre: Skipped 1 previous similar message [ 4712.928231] Lustre: 2365:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754410000/real 1754410000] req@ffff8a69f01fca80 x1839627434764160/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754410016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4712.929115] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 4712.937445] Lustre: 2365:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4712.943116] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba72dba2c to 0xc2e931ba72dc0bc [ 4713.138801] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4713.719522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4715.632674] Lustre: DEBUG MARKER: == sanity test 819a: too big niobuf in read ============== 12:06:58 (1754410018) [ 4718.774119] Lustre: DEBUG MARKER: == sanity test 819b: too big niobuf in write ============= 12:07:01 (1754410021) [ 4719.168210] LustreError: 2367:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -12 req@ffff8a69c7895c00 x1839627434772736/t0(0) o4->lustre-OST0000-osc-ffff8a69c7c6b000@192.168.201.141@tcp:6/4 lens 488/448 e 0 to 0 dl 1754410038 ref 2 fl Interpret:ReMQU/600/0 rc -12/-12 job:'dd.0' uid:0 gid:0 projid:0 [ 4719.174923] LustreError: 2367:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 9 previous similar messages [ 4719.200128] Lustre: 2364:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754410006/real 1754410006] req@ffff8a69f01ff100 x1839627434764544/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754410022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4723.798817] Lustre: DEBUG MARKER: == sanity test 820: update max EA from open intent ======= 12:07:06 (1754410026) [ 4724.402582] Lustre: DEBUG MARKER: SKIP: sanity test_820 needs >= 2 MDTs [ 4725.074505] Lustre: DEBUG MARKER: == sanity test 823: Setting create_count > OST_MAX_PRECREATE is lowered to maximum ========================================================== 12:07:07 (1754410027) [ 4727.679928] Lustre: DEBUG MARKER: setting create_count to 100200: [ 4728.369804] Lustre: DEBUG MARKER: -result- count: 9984 with max: 20000, expecting: 9984 [ 4731.607461] Lustre: DEBUG MARKER: == sanity test 831: throttling unlink/setattr queuing on OSP ========================================================== 12:07:14 (1754410034) [ 4739.493224] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69f70cd500 x1839627435316096/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410058 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4739.498726] Lustre: 221708:0:(client.c:1611:after_reply()) Skipped 2 previous similar messages [ 4745.703932] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69da1f7800 x1839627435343232/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410065 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4751.777859] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69f5c6c000 x1839627435344640/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410071 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4761.186040] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69c2fd0000 x1839627435397760/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410080 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4761.197431] Lustre: 221708:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 4770.474701] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69ffd80000 x1839627435425408/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410089 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4770.486252] Lustre: 221708:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 4792.290213] Lustre: 221708:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@ffff8a69ffd80380 x1839627435507072/t0(0) o36->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 488/456 e 0 to 0 dl 1754410111 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4792.301356] Lustre: 221708:0:(client.c:1611:after_reply()) Skipped 3 previous similar messages [ 4815.494478] Lustre: DEBUG MARKER: == sanity test 832: lfs rm_entry ========================= 12:08:38 (1754410118) [ 4816.021660] Lustre: DEBUG MARKER: SKIP: sanity test_832 needs >= 2 MDTs [ 4816.629901] Lustre: DEBUG MARKER: == sanity test 833: Mixed buffered/direct read and write should not return -EIO ========================================================== 12:08:39 (1754410119) [ 4856.744383] Lustre: DEBUG MARKER: SKIP: sanity test_842 skipping SLOW test 842 [ 4857.373386] Lustre: DEBUG MARKER: == sanity test 850: lljobstat can parse living and aggregated job_stats ========================================================== 12:09:20 (1754410160) [ 4860.151348] Lustre: DEBUG MARKER: == sanity test 851: fanotify can monitor open/read/write/close events for lustre fs ========================================================== 12:09:23 (1754410163) [ 4862.655988] Lustre: DEBUG MARKER: == sanity test 852: mkdir using intent lock for striped directory ========================================================== 12:09:25 (1754410165) [ 4863.205219] Lustre: DEBUG MARKER: SKIP: sanity test_852 needs >= 2 MDTs [ 4863.824471] Lustre: DEBUG MARKER: == sanity test 853: Verify that random fadvise works as expected ========================================================== 12:09:26 (1754410166) [ 4872.968473] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread ========================================================== 12:09:35 (1754410175) [ 4887.011326] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 192.168.201.141@tcp) was lost; in progress operations using this service will fail [ 4887.011466] Lustre: lustre-MDT0000-mdc-ffff8a69c7c6b000: Connection to lustre-MDT0000 (at 192.168.201.141@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4887.016544] Lustre: Evicted from MGS (at 192.168.201.141@tcp) after server handle changed from 0xc2e931ba72dc0bc to 0xc2e931ba72ee43f [ 4887.021850] Lustre: MGC192.168.201.141@tcp: Connection restored to 192.168.201.141@tcp (at 192.168.201.141@tcp) [ 4887.023958] Lustre: Skipped 1 previous similar message [ 4887.060343] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a69c7895880 x1839627434580608/t21474882100(21474882100) o101->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 576/608 e 0 to 0 dl 1754410206 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'md5sum.0' uid:0 gid:0 projid:0 [ 4887.067953] LustreError: 2363:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 4 previous similar messages [ 4888.544133] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4891.767398] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4892.334168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4893.152149] Lustre: 2367:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754410180/real 1754410180] req@ffff8a69c3d83480 x1839627435796096/t0(0) o400->lustre-MDT0000-mdc-ffff8a69c7c6b000@192.168.201.141@tcp:12/10 lens 224/224 e 0 to 1 dl 1754410196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4893.160932] Lustre: 2367:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4908.624128] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4908.632748] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4928.712105] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4928.721074] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 sleeping for 20000ms [ 4928.722394] LustreError: 226837:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69c7c6b000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4928.727735] LustreError: 226837:0:(lov_obd.c:784:lov_cleanup()) Skipped 5 previous similar messages [ 4928.732583] LustreError: 226837:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 4928.735636] LustreError: 226837:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4928.757761] Lustre: Unmounted lustre-client [ 4928.758992] Lustre: Skipped 2 previous similar messages [ 4948.792084] LustreError: 177545:0:(mgc_request.c:1780:mgc_process_log()) cfs_fail_timeout id 903 awake [ 4948.794938] LustreError: 177545:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4948.800159] LustreError: 226837:0:(obd_class.h:479:obd_check_dev()) Device 0 not setup [ 4948.803055] LustreError: 226837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4987.325492] Key type lgssc unregistered [ 4988.137949] LNet: 227296:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4988.167204] LNetError: 227296:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4988.203504] LNet: Removed LNI 192.168.201.41@tcp [ 4990.609441] Key type .llcrypt unregistered [ 4990.611205] Key type ._llcrypt unregistered [ 5007.924528] Key type ._llcrypt registered [ 5007.926993] Key type .llcrypt registered [ 5008.847429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 5008.895174] alg: No test for adler32 (adler32-zlib) [ 5011.049316] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5012.541138] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 5014.488190] Key type lgssc registered [ 5017.073714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5082.014582] Lustre: Mounted lustre-client [ 5085.465080] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5099.293378] Lustre: DEBUG MARKER: == sanity test 901: don't leak a mgc lock on client umount ========================================================== 12:13:21 (1754410401) [ 5101.617179] LustreError: 230388:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69e0a1c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5101.634907] LustreError: 230388:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 5101.683072] Lustre: Unmounted lustre-client [ 5102.038781] Lustre: Mounted lustre-client [ 5107.025243] Lustre: DEBUG MARKER: == sanity test 902: test short write doesn't hang lustre ========================================================== 12:13:29 (1754410409) [ 5107.235649] Lustre: *** cfs_fail_loc=2001415, val=0*** [ 5112.722586] Lustre: DEBUG MARKER: == sanity test 903: Test long page discard does not cause evictions ========================================================== 12:13:34 (1754410414) [ 5118.799871] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5138.873055] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5138.923244] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5159.024125] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5159.052462] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5179.080121] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5179.124862] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5199.160128] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5199.183333] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5219.208777] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5219.242149] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 sleeping for 20000ms [ 5239.248178] LustreError: 230413:0:(osc_cache.c:3385:osc_page_gang_lookup()) cfs_fail_timeout id 417 awake [ 5239.260558] LustreError: lustre-OST0000-osc-ffff8a69d8b0e000: operation ost_read to node 192.168.201.141@tcp failed: rc = -116 [ 5249.577322] Lustre: DEBUG MARKER: == sanity test 904: virtual project ID xattr ============= 12:15:51 (1754410551) [ 5250.938593] Lustre: DEBUG MARKER: SKIP: sanity test_904 ldiskfs only test [ 5252.587284] Lustre: DEBUG MARKER: == sanity test 905: bad or new opcode should not stuck client ========================================================== 12:15:54 (1754410554) [ 5253.827350] LustreError: lustre-OST0000-osc-ffff8a69d8b0e000: operation ost_ladvise to node 192.168.201.141@tcp failed: rc = -95 [ 5253.848482] LustreError: Skipped 1 previous similar message [ 5259.588540] Lustre: DEBUG MARKER: == sanity test 906: Simple test for io_uring I/O engine via fio ========================================================== 12:16:01 (1754410561) [ 5260.741230] Lustre: DEBUG MARKER: SKIP: sanity test_906 kernel does not support io_uring fully [ 5262.477865] Lustre: DEBUG MARKER: == sanity test 907: write rpc error during unlink ======== 12:16:04 (1754410564) [ 5264.516239] LustreError: lustre-OST0000-osc-ffff8a69d8b0e000: operation ost_write to node 192.168.201.141@tcp failed: rc = -3 [ 5264.520208] LustreError: Skipped 2 previous similar messages [ 5269.675796] Lustre: DEBUG MARKER: == sanity test 908a: llog created with valid ctime ======= 12:16:11 (1754410571) [ 5270.845413] Lustre: DEBUG MARKER: SKIP: sanity test_908a ldiskfs only test [ 5272.193226] Lustre: DEBUG MARKER: == sanity test 908b: changelog stores valid mtime ======== 12:16:14 (1754410574) [ 5273.452579] Lustre: DEBUG MARKER: SKIP: sanity test_908b ldiskfs only test [ 5274.766802] Lustre: DEBUG MARKER: == sanity test complete, duration 5120 sec =============== 12:16:17 (1754410577) [ 5276.066480] Lustre: DEBUG MARKER: === sanity: start cleanup 12:16:18 (1754410578) === [ 5294.702421] Lustre: DEBUG MARKER: === sanity: finish cleanup 12:16:36 (1754410596) === [ 5295.745738] LustreError: 237756:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff8a69d8b0e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5295.750319] LustreError: 237756:0:(lov_obd.c:784:lov_cleanup()) Skipped 1 previous similar message [ 5295.763960] LustreError: 237756:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 5295.767516] LustreError: 237756:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5295.798845] Lustre: Unmounted lustre-client [ 5327.989284] Key type lgssc unregistered [ 5328.220714] LNet: 238235:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5328.225254] LNetError: 238235:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5328.245753] LNet: Removed LNI 192.168.201.41@tcp [ 5328.645148] Key type .llcrypt unregistered [ 5328.647330] Key type ._llcrypt unregistered