[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 668943308 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002294] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.008382] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010008] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011025] pid_max: default: 32768 minimum: 301 [ 0.012268] LSM: Security Framework initializing [ 0.013084] Yama: becoming mindful. [ 0.014067] SELinux: Initializing. [ 0.015139] *** VALIDATE selinux *** [ 0.025000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029442] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030224] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032176] *** VALIDATE tmpfs *** [ 0.034490] *** VALIDATE proc *** [ 0.036327] *** VALIDATE cgroup *** [ 0.037021] *** VALIDATE cgroup2 *** [ 0.038313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040048] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042034] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046603] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048901] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049035] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052020] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054026] ... fixed-purpose events: 3 [ 0.055029] ... event mask: 000000070000000f [ 0.056405] rcu: Hierarchical SRCU implementation. [ 0.058631] smp: Bringing up secondary CPUs ... [ 0.059592] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.073106] smp: Brought up 1 node, 4 CPUs [ 0.075018] smpboot: Max logical packages: 1 [ 0.076011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.115013] node 0 deferred pages initialised in 30ms [ 0.118443] devtmpfs: initialized [ 0.119335] x86/mm: Memory block size: 128MB [ 0.122456] gcov: version magic: 0x41383552 [ 0.125352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130333] pinctrl core: initialized pinctrl subsystem [ 0.132209] [ 0.132799] ************************************************************* [ 0.135015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137016] ** ** [ 0.140016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142019] ** ** [ 0.144017] ** This means that this kernel is built to expose internal ** [ 0.146023] ** IOMMU data structures, which may compromise security on ** [ 0.148011] ** your system. ** [ 0.151013] ** ** [ 0.153012] ** If you see this message and you are not debugging the ** [ 0.156014] ** kernel, report this immediately to your vendor! ** [ 0.158008] ** ** [ 0.160055] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162021] ************************************************************* [ 0.164838] NET: Registered protocol family 16 [ 0.166489] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176104] cpuidle: using governor menu [ 0.178008] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182481] PCI: Using configuration type 1 for base access [ 0.184154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192193] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196182] cryptd: max_cpu_qlen set to 1000 [ 0.197305] ACPI: Added _OSI(Module Device) [ 0.198000] ACPI: Added _OSI(Processor Device) [ 0.199041] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206675] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213820] ACPI: Interpreter enabled [ 0.215077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217015] ACPI: Using IOAPIC for interrupt routing [ 0.219265] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220347] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234535] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237089] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243161] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249760] acpiphp: Slot [2] registered [ 0.250119] acpiphp: Slot [5] registered [ 0.252119] acpiphp: Slot [6] registered [ 0.253125] acpiphp: Slot [7] registered [ 0.255113] acpiphp: Slot [8] registered [ 0.256112] acpiphp: Slot [9] registered [ 0.258098] acpiphp: Slot [10] registered [ 0.259101] acpiphp: Slot [3] registered [ 0.261087] acpiphp: Slot [4] registered [ 0.262087] acpiphp: Slot [11] registered [ 0.263129] acpiphp: Slot [12] registered [ 0.265115] acpiphp: Slot [13] registered [ 0.266119] acpiphp: Slot [14] registered [ 0.268128] acpiphp: Slot [15] registered [ 0.269160] acpiphp: Slot [16] registered [ 0.270119] acpiphp: Slot [17] registered [ 0.272125] acpiphp: Slot [18] registered [ 0.274140] acpiphp: Slot [19] registered [ 0.275170] acpiphp: Slot [20] registered [ 0.277163] acpiphp: Slot [21] registered [ 0.279209] acpiphp: Slot [22] registered [ 0.280110] acpiphp: Slot [23] registered [ 0.282175] acpiphp: Slot [24] registered [ 0.284267] acpiphp: Slot [25] registered [ 0.285122] acpiphp: Slot [26] registered [ 0.287195] acpiphp: Slot [27] registered [ 0.288122] acpiphp: Slot [28] registered [ 0.289249] acpiphp: Slot [29] registered [ 0.290000] acpiphp: Slot [30] registered [ 0.290000] acpiphp: Slot [31] registered [ 0.292066] PCI host bridge to bus 0000:00 [ 0.294061] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.296031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313377] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317399] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327909] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332662] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343597] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.345902] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.349035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.351885] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358054] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383570] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.493013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.500015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541869] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.557028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583048] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.611015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.630483] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.651014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.701015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.727203] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.744016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.754017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.787021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.805021] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.821173] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.833014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.868015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.883706] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.909448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.912368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.915377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.918224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.923114] iommu: Default domain type: Passthrough [ 0.926420] SCSI subsystem initialized [ 0.927076] ACPI: bus type USB registered [ 0.928082] usbcore: registered new interface driver usbfs [ 0.930083] usbcore: registered new interface driver hub [ 0.932072] usbcore: registered new device driver usb [ 0.933190] pps_core: LinuxPPS API ver. 1 registered [ 0.935008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.939119] PTP clock support registered [ 0.941442] EDAC MC: Ver: 3.0.0 [ 0.943997] PCI: Using ACPI for IRQ routing [ 0.946489] NetLabel: Initializing [ 0.948012] NetLabel: domain hash size = 128 [ 0.950011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.952210] NetLabel: unlabeled traffic allowed by default [ 0.955348] vgaarb: loaded [ 0.957705] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.960011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.965410] clocksource: Switched to clocksource kvm-clock [ 1.157245] VFS: Disk quotas dquot_6.6.0 [ 1.159544] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.162741] *** VALIDATE ramfs *** [ 1.164610] *** VALIDATE hugetlbfs *** [ 1.166774] pnp: PnP ACPI init [ 1.171461] pnp: PnP ACPI: found 6 devices [ 1.190820] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.195355] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.198555] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.200885] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.203816] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.206193] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.209378] NET: Registered protocol family 2 [ 1.213526] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.223403] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.227868] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.236473] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.242960] TCP: Hash tables configured (established 65536 bind 65536) [ 1.247363] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.251629] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.254818] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.258847] NET: Registered protocol family 1 [ 1.263344] RPC: Registered named UNIX socket transport module. [ 1.265205] RPC: Registered udp transport module. [ 1.266889] RPC: Registered tcp transport module. [ 1.268536] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.270961] NET: Registered protocol family 44 [ 1.272461] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.274392] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.276672] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.279210] PCI: CLS 0 bytes, default 64 [ 1.281757] Unpacking initramfs... [ 3.297573] debug: unmapping init [mem 0xffff8a097cc54000-0xffff8a097ffbffff] [ 3.302921] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.305425] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.309337] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.852250] Initialise system trusted keyrings [ 3.854313] Key type blacklist registered [ 3.856611] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.867813] zbud: loaded [ 3.871402] *** VALIDATE nfs *** [ 3.872839] *** VALIDATE nfs4 *** [ 3.874978] pstore: using deflate compression [ 3.879360] Platform Keyring initialized [ 4.039513] NET: Registered protocol family 38 [ 4.041318] Key type asymmetric registered [ 4.043434] Asymmetric key parser 'x509' registered [ 4.045134] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.048043] io scheduler mq-deadline registered [ 4.049543] io scheduler kyber registered [ 4.051401] io scheduler bfq registered [ 4.053430] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.056621] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.059988] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.063318] ACPI: Power Button [PWRF] [ 4.070465] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.077885] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.097766] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.110702] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.135964] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.168948] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.204895] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.209415] Non-volatile memory driver v1.3 [ 4.210887] Linux agpgart interface v0.103 [ 4.243851] virtio_blk virtio1: [vda] 134152 512-byte logical blocks (68.7 MB/65.5 MiB) [ 4.246437] vda: detected capacity change from 0 to 68685824 [ 4.287462] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.290350] vdb: detected capacity change from 0 to 1073741824 [ 4.313090] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.315486] vdc: detected capacity change from 0 to 2621440000 [ 4.329860] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.332656] vdd: detected capacity change from 0 to 2621440000 [ 4.347567] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.350179] vde: detected capacity change from 0 to 4294967296 [ 4.367315] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.370650] vdf: detected capacity change from 0 to 4294967296 [ 4.377831] libphy: Fixed MDIO Bus: probed [ 4.384748] usbcore: registered new interface driver usbserial_generic [ 4.387862] usbserial: USB Serial support registered for generic [ 4.390508] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.395430] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.397361] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.400330] mousedev: PS/2 mouse device common for all mice [ 4.403349] rtc_cmos 00:05: RTC can wake from S4 [ 4.407966] rtc_cmos 00:05: registered as rtc0 [ 4.410626] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.413467] intel_pstate: CPU model not supported [ 4.416452] hid: raw HID events driver (C) Jiri Kosina [ 4.418290] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.422642] usbcore: registered new interface driver usbhid [ 4.426953] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.431325] usbhid: USB HID core driver [ 4.432788] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.437191] drop_monitor: Initializing network drop monitor service [ 4.439595] Initializing XFRM netlink socket [ 4.441691] NET: Registered protocol family 10 [ 4.444350] Segment Routing with IPv6 [ 4.445443] NET: Registered protocol family 17 [ 4.447434] mpls_gso: MPLS GSO support [ 4.452327] RAS: Correctable Errors collector initialized. [ 4.453968] AVX version of gcm_enc/dec engaged. [ 4.455513] AES CTR mode by8 optimization enabled [ 4.544753] sched_clock: Marking stable (4544673253, 0)->(5763624391, -1218951138) [ 4.549347] registered taskstats version 1 [ 4.552425] Loading compiled-in X.509 certificates [ 4.555464] zswap: loaded using pool lzo/zbud [ 4.589542] Key type big_key registered [ 4.606335] Key type encrypted registered [ 4.609696] ima: No TPM chip found, activating TPM-bypass! [ 4.611980] ima: Allocated hash algorithm: sha1 [ 4.613493] ima: No architecture policies found [ 4.615129] evm: Initialising EVM extended attributes: [ 4.618489] evm: security.selinux [ 4.619699] evm: security.ima [ 4.620845] evm: security.capability [ 4.622236] evm: HMAC attrs: 0x1 [ 4.626342] rtc_cmos 00:05: setting system clock to 2026-05-09 00:24:48 UTC (1778286288) [ 4.636499] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 4.639771] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 4.649155] Write protecting the kernel read-only data: 28672k [ 4.653574] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 4.656482] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 4.720704] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.735879] systemd[1]: Detected virtualization kvm. [ 4.739712] systemd[1]: Detected architecture x86-64. [ 4.744431] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.787537] systemd[1]: No hostname configured. [ 4.789891] systemd[1]: Set hostname to . [ 4.792749] random: systemd: uninitialized urandom read (16 bytes read) [ 4.870212] systemd[1]: Initializing machine ID from random generator. [ 5.117800] random: ln: uninitialized urandom read (6 bytes read) [ 5.346331] random: systemd: uninitialized urandom read (16 bytes read) [ 5.351194] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.366821] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.381315] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.453219] device-mapper: uevent: version 1.0.3 [ 7.455445] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.377304] virtio_net virtio0 ens2: renamed from eth0 [ 10.394589] random: fast init done [ 10.731978] scsi host0: ata_piix [ 10.886639] scsi host1: ata_piix [ 10.889892] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.895891] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.552980] random: crng init done [ 15.561213] random: 7 urandom warning(s) missed due to ratelimiting [ 18.497755] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.301564] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.542037] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.434302] SELinux: Disabled at runtime. [ 25.561347] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.584615] systemd[1]: Detected virtualization kvm. [ 25.588583] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.355332] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.361839] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.397784] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.411905] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.418614] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.476116] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.504219] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target Slices. [ 27.876383] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 29.619307] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.856228] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.981642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.018533] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.211781] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 39.384812] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 40.305640] NFS: Registering the id_resolver key type [ 40.307292] Key type id_resolver registered [ 40.311345] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ 49.601005] hrtimer: interrupt took 7997144 ns [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 101.422779] libcfs: loading out-of-tree module taints kernel. [ 101.459505] Key type ._llcrypt registered [ 101.462503] Key type .llcrypt registered [ 101.559760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 118.922596] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 120.229567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.253945] alg: No test for adler32 (adler32-zlib) [ 121.690533] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 122.619295] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 124.407171] Key type lgssc registered [ 126.484927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.852268] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 183.787675] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 196.527383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.552844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.928881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.970435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.136573] Lustre: lustre-MDT0000: new disk, initializing [ 198.225026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.261163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.767966] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.708261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.879374] Lustre: 6503:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.928261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.933807] Lustre: Skipped 1 previous similar message [ 217.111750] Lustre: lustre-MDT0001: new disk, initializing [ 217.267745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.335295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.364894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.824737] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.939947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.216617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.462243] Lustre: lustre-OST0000: new disk, initializing [ 240.466501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.540236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.799540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.884665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.901891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 249.030254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.231122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.368758] Lustre: lustre-OST0001: new disk, initializing [ 263.371697] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.435705] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.362301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.371603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.464485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.952521] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.053142] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.212480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.539816] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.134498] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 311.138392] Lustre: DEBUG MARKER: Client: 2.17.52.6 [ 314.116150] Lustre: DEBUG MARKER: MDS: 2.17.52.6 [ 317.123564] Lustre: DEBUG MARKER: OSS: 2.17.52.6 [ 319.302753] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Fri May 8 20:30:01 EDT 2026 [ 339.548328] Lustre: DEBUG MARKER: - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34681862 <= 34618625) for LU-18938, skip 360 [ 341.183568] Lustre: DEBUG MARKER: - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34681862 < 34486116) for LU-14927, skip 0f [ 342.758458] Lustre: DEBUG MARKER: excepting tests: 42a 42c 42b 118c 118d 407 119i 817 411a [ 344.369571] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 [ 345.824219] Lustre: DEBUG MARKER: === sanity: start setup 20:30:29 (1778286629) === [ 349.684557] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing check_config_client /mnt/lustre [ 370.998882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 374.042622] Lustre: 13320:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 377.891507] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 383.720972] Lustre: DEBUG MARKER: === sanity: finish setup 20:31:06 (1778286666) === [ 390.099494] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 20:31:12 (1778286672) [ 393.314849] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 395.387703] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 20:31:18 (1778286678) [ 403.561576] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 20:31:26 (1778286686) [ 611.230644] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 20:34:54 (1778286894) [ 618.724207] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 20:35:01 (1778286901) [ 620.282857] Lustre: *** cfs_fail_loc=15b, val=0*** [ 620.287923] Lustre: *** cfs_fail_loc=15b, val=0*** [ 620.291902] LustreError: 11281:0:(llog_cat.c:583:llog_cat_add_rec()) lustre-OST0000-osc-MDT0000: initialization error: rc = -28 [ 627.903256] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 20:35:10 (1778286910) [ 636.914663] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 20:35:19 (1778286919) [ 638.242241] Lustre: *** cfs_fail_loc=19a, val=0*** [ 639.202429] Lustre: *** cfs_fail_loc=19a, val=0*** [ 639.209795] ------------[ cut here ]------------ [ 639.217632] do not call blocking ops when !TASK_RUNNING; state=402 set at [<000000008d2baa48>] distribute_txn_commit_thread+0x95/0x1000 [ptlrpc] [ 639.232240] WARNING: CPU: 1 PID: 6540 at kernel/sched/core.c:7471 __might_sleep+0x9d/0xc0 [ 639.237611] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix serio_raw ghash_clmulni_intel libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 639.267203] CPU: 1 PID: 6540 Comm: dist_txn-0 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 639.274351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 639.277900] RIP: 0010:__might_sleep+0x9d/0xc0 [ 639.280282] Code: a8 12 00 00 48 c7 c7 a8 56 78 a1 48 83 05 ba 89 df 02 01 c6 05 2f 83 52 02 01 48 89 d1 e8 a7 50 fb ff 48 83 05 ab 89 df 02 01 <0f> 0b 48 83 05 a9 89 df 02 01 48 83 05 a9 89 df 02 01 eb 9b 66 66 [ 639.298128] RSP: 0018:ffffaa9381dbbd78 EFLAGS: 00010202 [ 639.302807] RAX: 0000000000000000 RBX: ffffffffa17add7d RCX: 0000000000000000 [ 639.311159] RDX: ffff8a0a020ae640 RSI: ffff8a0a0209e5a8 RDI: ffff8a0a0209e5a8 [ 639.315277] RBP: 00000000000000e2 R08: 0000000000000000 R09: c0000000ffff7fff [ 639.322596] R10: 0000000000000001 R11: ffffaa9381dbbb68 R12: 0000000000000000 [ 639.329971] R13: 0000000000000001 R14: 0000000000000058 R15: ffffffffc0e18238 [ 639.332929] FS: 0000000000000000(0000) GS:ffff8a0a02080000(0000) knlGS:0000000000000000 [ 639.339613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 639.342411] CR2: 00005607730b01c8 CR3: 00000000a1016006 CR4: 0000000000170ee0 [ 639.346952] Call Trace: [ 639.348676] ? show_regs.cold.9+0x22/0x2f [ 639.351698] ? __warn+0xc8/0x150 [ 639.353088] ? __might_sleep+0x9d/0xc0 [ 639.357821] ? report_bug+0x113/0x140 [ 639.360811] ? do_error_trap+0xb6/0x130 [ 639.364558] ? do_invalid_op+0x46/0x60 [ 639.365805] ? __might_sleep+0x9d/0xc0 [ 639.367059] ? invalid_op+0x14/0x20 [ 639.369181] ? distribute_txn_commit_batchid_update+0x68/0xa00 [ptlrpc] [ 639.374056] ? __might_sleep+0x9d/0xc0 [ 639.378348] ? __might_sleep+0x95/0xc0 [ 639.382268] slab_pre_alloc_hook.constprop.64+0x11f/0x1d0 [ 639.388928] kmem_cache_alloc_trace+0x5b/0x440 [ 639.392266] ? top_multiple_thandle_destroy+0x3a2/0x420 [ptlrpc] [ 639.397273] distribute_txn_commit_batchid_update+0x68/0xa00 [ptlrpc] [ 639.401279] distribute_txn_commit_thread+0xa97/0x1000 [ptlrpc] [ 639.407487] ? distribute_txn_commit_batchid_update+0xa00/0xa00 [ptlrpc] [ 639.415307] kthread+0x1d1/0x200 [ 639.416441] ? set_kthread_struct+0x70/0x70 [ 639.420405] ret_from_fork+0x1f/0x30 [ 639.424347] ---[ end trace 8ff8e2d705defe2e ]--- [ 641.069454] Lustre: *** cfs_fail_loc=19a, val=0*** [ 641.071397] Lustre: Skipped 1 previous similar message [ 643.145466] Lustre: *** cfs_fail_loc=19a, val=0*** [ 643.152462] Lustre: Skipped 1 previous similar message [ 648.150959] Lustre: *** cfs_fail_loc=19a, val=0*** [ 648.157371] Lustre: Skipped 4 previous similar messages [ 657.007878] Lustre: *** cfs_fail_loc=19a, val=0*** [ 657.021262] Lustre: Skipped 8 previous similar messages [ 673.158285] Lustre: *** cfs_fail_loc=19a, val=0*** [ 673.160372] Lustre: Skipped 15 previous similar messages [ 706.047432] Lustre: *** cfs_fail_loc=19a, val=0*** [ 706.051564] Lustre: Skipped 29 previous similar messages [ 763.451688] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 20:37:25 (1778287045) [ 765.141329] Lustre: *** cfs_fail_loc=188, val=0*** [ 772.048458] Lustre: *** cfs_fail_loc=189, val=0*** [ 788.693405] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 791.792749] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 20:37:53 (1778287073) [ 799.489127] Lustre: lustre-MDD0000: changelog on [ 803.236881] Lustre: lustre-MDD0001: changelog on [ 813.759904] Lustre: DEBUG MARKER: SKIP: sanity test_60j path oi.1/0x1:0xb:0x0 is not in 'O/1/d/' format [ 816.946494] Lustre: lustre-MDD0001: changelog off [ 819.993750] Lustre: lustre-MDD0000: changelog off [ 822.898269] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 20:38:25 (1778287105) [ 830.967660] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 20:38:33 (1778287113) [ 838.666858] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 20:38:41 (1778287121) [ 909.215248] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 20:39:52 (1778287192) [ 922.997725] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 20:40:05 (1778287205) [ 933.088199] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 934.746584] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 20:40:17 (1778287217) [ 947.552507] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 20:40:29 (1778287229) [ 1007.273526] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 20:41:30 (1778287290) [ 1011.689629] Lustre: *** cfs_fail_loc=725, val=0*** [ 1019.754765] Lustre: *** cfs_fail_loc=725, val=0*** [ 1028.663708] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 20:41:51 (1778287311) [ 1043.291740] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 20:42:06 (1778287326) [ 1100.540699] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 20:43:02 (1778287382) [ 1125.428770] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 20:43:28 (1778287408) [ 1135.482691] Lustre: *** cfs_fail_loc=513, val=17*** [ 1135.484440] LustreError: 8388:0:(service.c:2320:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1864668435893888 [ 1137.543996] Lustre: Failing over lustre-OST0000 [ 1137.635345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.909308] Lustre: server umount lustre-OST0000 complete [ 1142.752359] LustreError: 8389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.780710] LustreError: 8389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1147.876775] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.898651] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1152.995711] LustreError: 8390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.032757] LustreError: 8390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1157.758354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1158.119136] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 1158.145372] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1159.710504] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1159.963684] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1159.965659] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 1165.308310] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [ 1173.615895] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1175.584525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1188.590832] Lustre: DEBUG MARKER: == sanity test 64j: check grants on re-done rpc ========== 20:44:31 (1778287471) [ 1190.204870] Lustre: *** cfs_fail_loc=256, val=0*** [ 1200.448911] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 20:44:43 (1778287483) [ 1209.223925] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 20:44:51 (1778287491) [ 1218.011496] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 20:45:00 (1778287500) [ 1227.701647] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 20:45:10 (1778287510) [ 1236.518672] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 20:45:19 (1778287519) [ 1245.777865] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 20:45:28 (1778287528) [ 1254.191683] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 20:45:36 (1778287536) [ 1263.687931] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 20:45:46 (1778287546) [ 1272.417536] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 20:45:55 (1778287555) [ 1283.240674] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 20:46:05 (1778287565) [ 1292.254347] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 20:46:14 (1778287574) [ 1294.985752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.995422] Lustre: Skipped 1 previous similar message [ 1295.001309] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1295.007892] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1295.024221] Lustre: Skipped 1 previous similar message [ 1296.167248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.187625] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1296.205849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1297.274625] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1298.332127] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.345991] Lustre: Skipped 1 previous similar message [ 1298.360670] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1298.368051] Lustre: Skipped 1 previous similar message [ 1323.722427] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1345.537825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.565232] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1345.577237] Lustre: Skipped 1 previous similar message [ 1345.580935] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1345.594814] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1350.532734] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1350.810766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1356.153384] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1356.379755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1379.784392] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 1401.033759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.050084] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1401.058901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1401.078866] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1406.175737] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1406.471131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1411.313200] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1411.566534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1437.365705] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1456.038921] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.050770] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 1456.057840] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1456.069387] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1460.436580] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1460.714371] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1464.734500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1464.974514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1487.586213] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1503.563026] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.572996] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1503.579946] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1503.589479] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 1507.369902] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1507.597042] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1511.192953] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1511.411942] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1517.762730] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 20:50:00 (1778287800) [ 1524.491314] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 20:50:07 (1778287807) [ 1530.861785] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 20:50:13 (1778287813) [ 1563.355128] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 20:50:46 (1778287846) [ 1592.917814] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 20:51:15 (1778287875) [ 1600.987854] Lustre: DEBUG MARKER: == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 20:51:23 (1778287883) [ 1608.865935] Lustre: DEBUG MARKER: == sanity test 65r: prevent all-zero offsets ============= 20:51:31 (1778287891) [ 1617.272189] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 20:51:40 (1778287900) [ 1631.896543] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 20:51:54 (1778287914) [ 1633.769609] Lustre: *** cfs_fail_loc=217, val=0*** [ 1633.775789] Lustre: Skipped 1 previous similar message [ 1636.715595] Lustre: *** cfs_fail_loc=217, val=0*** [ 1644.902938] Lustre: DEBUG MARKER: == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 20:52:07 (1778287927) [ 1660.468487] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 1662.411725] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 20:52:25 (1778287945) [ 1670.102437] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 20:52:33 (1778287953) [ 1678.747304] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 20:52:41 (1778287961) [ 1713.939165] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 20:53:16 (1778287996) [ 1720.834797] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 20:53:23 (1778288003) [ 1727.678527] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 20:53:30 (1778288010) [ 1734.034644] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 20:53:37 (1778288017) [ 1824.930324] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 20:55:07 (1778288107) [ 1884.445944] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 20:56:07 (1778288167) [ 1894.789846] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 20:56:17 (1778288177) [ 1895.638746] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303]: client csum 581e973, server csum 581e972 [ 1899.383343] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1901.004440] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum f577db96 (type 1), server csum e7de698c (type 1) [ 1904.402507] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1905.656345] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum 6b176c68 (type 2), server csum 13846d62 (type 2) [ 1908.530797] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1910.134289] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum e425fc53 (type 4), server csum 80c9bc4f (type 4) [ 1912.852646] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1914.424804] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum e522fde2 (type 10), server csum 25c8fde7 (type 10) [ 1916.952659] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1920.923667] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1922.619958] LustreError: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc20:0x0] object 0x2c0000401:3829 extent [0-4194303], client returned csum 1b2ce449 (type 40), server csum 5b82e486 (type 40) [ 1922.646216] LustreError: Skipped 1 previous similar message [ 1925.432271] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1929.394715] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1936.315338] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 20:56:59 (1778288219) [ 1942.242938] Lustre: 8393:0:(tgt_handler.c:1962:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc21:0x0]:[0-1048575]-5884ef7b-630cef80 [ 1942.261437] LustreError: dumping log to /tmp/lustre-log.1778288226.8393 [ 1942.362739] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc21:0x0] object 0x280000401:3841 extent [0-1048575], client returned csum 5884ef7b (type 20), server csum 630cef80 (type 20) [ 1942.374277] LustreError: Skipped 1 previous similar message [ 1971.544502] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 20:57:34 (1778288254) [ 1972.090170] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc23:0x0] object 0x280000401:3842 extent [0-4194303]: client csum fec05011, server csum fec05010 [ 1974.993000] LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.204.6@tcp inode [0x200000407:0xc23:0x0] object 0x280000401:3842 extent [4194304-8388607], client returned csum 872950a1 (type 20), server csum fec05010 (type 20) [ 1982.978698] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 20:57:45 (1778288265) [ 1985.218794] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1985.673527] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1989.208270] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1989.221356] LustreError: Skipped 3 previous similar messages [ 1996.766470] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 1996.798442] LustreError: Skipped 3 previous similar messages [ 2009.155171] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 2009.180020] LustreError: Skipped 3 previous similar messages [ 2033.718889] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 2033.739992] LustreError: Skipped 5 previous similar messages [ 2046.681810] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2069.543691] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 2069.561408] LustreError: Skipped 15 previous similar messages [ 2106.702611] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2136.160911] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e [ 2136.189637] LustreError: Skipped 24 previous similar messages [ 2166.976056] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2227.918878] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2267.188882] LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.204.6@tcp inode [0x200000407:0xc24:0x0] object 0x2c0000401:3830 extent [4194304-8388607]: client csum fec05011, server csum fec05010 [ 2267.215077] LustreError: Skipped 44 previous similar messages [ 2287.786272] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2347.421213] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2408.737841] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2417.375372] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 21:04:59 (1778288699) [ 2419.798458] Lustre: *** cfs_fail_loc=21a, val=0*** [ 2423.386609] Lustre: *** cfs_fail_loc=21b, val=0*** [ 2434.971448] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 21:05:17 (1778288717) [ 2436.166403] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 2442.922857] Lustre: Modifying parameter lustre.osc.lustre*.checksums=1 in log params [ 2448.592813] Lustre: Disabling parameter lustre.osc.lustre*.checksums= in log params [ 2465.031747] Lustre: Setting parameter lustre.osc.lustre*.checksums=0 in log params [ 2469.524459] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 21:05:52 (1778288752) [ 2471.812453] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 2473.667178] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2478.273195] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2480.066506] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 0 sec [ 2484.168926] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2485.820762] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2487.825423] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2492.451861] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2501.376497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 7 sec [ 2505.798147] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2507.351403] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2509.149221] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2513.474572] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2521.760619] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 6 sec [ 2525.969984] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2527.720840] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2529.700863] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2534.136577] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2542.158948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 6 sec [ 2546.479823] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2548.076337] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2549.740839] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2553.744996] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2562.835518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 7 sec [ 2566.496240] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2567.823347] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2569.524394] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2573.597450] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2582.621169] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 7 sec [ 2586.722935] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2588.535433] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2590.330100] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2594.290843] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2603.455530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in IDLE state after 7 sec [ 2606.824308] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid 50 [ 2608.332495] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5493ab000.ost_server_uuid in FULL state after 0 sec [ 2615.131044] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2616.665180] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 21:08:19 (1778288899) [ 2623.163608] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 21:08:26 (1778288906) [ 2625.614781] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2627.127413] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2628.759624] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 2630.703313] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 2638.462804] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2640.330452] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 21:08:43 (1778288923) [ 2651.895313] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 21:08:54 (1778288934) [ 2661.959483] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check =========== 21:09:04 (1778288944) [ 2677.918700] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 21:09:21 (1778288961) [ 2686.361740] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 21:09:29 (1778288969) [ 2687.944210] Lustre: *** cfs_fail_loc=228, val=0*** [ 2694.272797] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 21:09:37 (1778288977) [ 2695.750987] Lustre: *** cfs_fail_loc=228, val=0*** [ 2702.167966] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 21:09:45 (1778288985) [ 2728.187278] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 21:10:11 (1778289011) [ 2735.466425] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 21:10:18 (1778289018) [ 2940.938691] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 21:13:43 (1778289223) [ 2973.578138] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 21:14:16 (1778289256) [ 3053.183345] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 21:15:36 (1778289336) [ 3347.479937] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 21:20:30 (1778289630) [ 3430.056160] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 21:21:53 (1778289713) [ 3437.636589] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 21:22:00 (1778289720) [ 3495.731284] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 21:22:59 (1778289779) [ 3511.524630] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 21:23:14 (1778289794) [ 3519.461030] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 21:23:22 (1778289802) [ 3575.384727] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 21:24:18 (1778289858) [ 3585.764151] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 21:24:29 (1778289869) [ 3591.458048] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 21:24:34 (1778289874) [ 3599.726732] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 21:24:43 (1778289883) [ 3605.160224] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 21:24:48 (1778289888) [ 3620.820286] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 21:25:04 (1778289904) [ 3635.857505] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 21:25:19 (1778289919) [ 3637.440136] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 3638.656457] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 3639.930552] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 3641.228850] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3646.203705] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 21:25:29 (1778289929) [ 3648.712905] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 3649.963511] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 3651.130393] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 3652.365165] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3654.036077] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 3659.444947] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 21:25:42 (1778289942) [ 3665.136349] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 21:25:48 (1778289948) [ 3681.103878] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 21:26:04 (1778289964) [ 3687.315677] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 21:26:10 (1778289970) [ 3692.578608] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 21:26:15 (1778289975) [ 3697.826849] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 21:26:21 (1778289981) [ 3703.878428] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 21:26:27 (1778289987) [ 3708.951678] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 21:26:32 (1778289992) [ 3714.042106] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 21:26:37 (1778289997) [ 3719.947231] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 21:26:43 (1778290003) [ 3725.759480] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 21:26:49 (1778290009) [ 3732.257829] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 21:26:55 (1778290015) [ 3956.119908] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 21:30:39 (1778290239) [ 4043.289605] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 21:32:06 (1778290326) [ 4046.897711] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 21:32:10 (1778290330) [ 4571.254029] Lustre: 22440:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30558 > trans_max 3200 [ 4571.257749] Lustre: 22440:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 4571.261386] Lustre: 22440:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 4571.265538] Lustre: 22440:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/151/0, punch: 0/0/0, quota 0/0/0 [ 4571.270759] Lustre: 22440:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 4571.273824] Lustre: 22440:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 4571.277933] CPU: 3 PID: 22440 Comm: mdt00_005 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 4571.281252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 4571.284110] Call Trace: [ 4571.285098] ? dump_stack+0xbb/0x10e [ 4571.286153] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 4571.288351] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 4571.290433] ? lod_trans_start+0x109/0x4c0 [lod] [ 4571.292175] ? mdd_trans_start+0x18/0x30 [mdd] [ 4571.293627] ? mdd_unlink+0x758/0x1350 [mdd] [ 4571.295173] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 4571.296657] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 4571.298196] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 4571.300504] ? mdt_reint+0x163/0x190 [mdt] [ 4571.303029] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 4571.304982] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [ 4571.306931] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 4571.309276] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 4571.311684] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 4571.313422] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 4571.315349] ? kthread+0x1d1/0x200 [ 4571.316421] ? set_kthread_struct+0x70/0x70 [ 4571.317904] ? ret_from_fork+0x1f/0x30 [ 4574.772060] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 21:40:58 (1778290858) [ 4576.412873] Lustre: lustre-MDD0000: changelog on [ 4577.685780] Lustre: lustre-MDD0001: changelog on [ 4581.748470] Lustre: lustre-MDD0001: changelog off [ 4582.752744] Lustre: lustre-MDD0000: changelog off [ 4583.834534] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 21:41:07 (1778290867) [ 4584.053672] Lustre: lustre-OST0000: Client 3edb92d3-9655-4671-8096-eb4da2e6a266 (at 192.168.204.6@tcp) reconnecting [ 4585.548337] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93d5527e7800.ost_server_uuid 50 [ 4586.306458] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93d5527e7800.ost_server_uuid in FULL state after 0 sec [ 4589.285073] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 21:41:13 (1778290873) [ 4592.365701] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 21:41:16 (1778290876) [ 4593.104607] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 4593.952776] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 21:41:17 (1778290877) [ 4597.153576] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 21:41:20 (1778290880) [ 4600.211914] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 21:41:24 (1778290884) [ 4603.013170] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 21:41:26 (1778290886) [ 4605.814547] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 21:41:29 (1778290889) [ 4619.262649] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 21:41:43 (1778290903) [ 4622.129613] Lustre: DEBUG MARKER: == sanity test 105f: Enqueue same range flocks =========== 21:41:45 (1778290905) [ 4625.877642] Lustre: DEBUG MARKER: == sanity test 105g: ldlm_lock_debug stack test ========== 21:41:49 (1778290909) [ 4630.713479] Lustre: DEBUG MARKER: == sanity test 105h: Flock functional verify ============= 21:41:54 (1778290914) [ 4633.733442] Lustre: DEBUG MARKER: == sanity test 105i: Flock deadlock verify =============== 21:41:57 (1778290917) [ 4641.497799] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 21:42:05 (1778290925) [ 4644.522991] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 21:42:08 (1778290928) [ 4649.017507] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 21:42:12 (1778290932) [ 4652.221106] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 21:42:16 (1778290936) [ 4743.856481] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 21:43:47 (1778291027) [ 4745.056994] Lustre: *** cfs_fail_loc=147, val=0*** [ 4748.746032] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 21:43:52 (1778291032) [ 4751.717263] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 21:43:55 (1778291035) [ 4754.899293] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 21:43:58 (1778291038) [ 4755.800216] Lustre: *** cfs_fail_loc=217, val=0*** [ 4755.801941] Lustre: Skipped 35 previous similar messages [ 4759.065068] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 4759.771124] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 4760.534844] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 21:44:04 (1778291044) [ 4763.827777] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 21:44:07 (1778291047) [ 4766.994306] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 21:44:10 (1778291050) [ 4767.846278] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4768.926636] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4770.784595] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4773.856884] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4777.891685] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4781.326338] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 21:44:25 (1778291065) [ 4791.057748] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 21:44:34 (1778291074) [ 4791.919149] Lustre: *** cfs_fail_loc=220, val=0*** [ 4791.921218] Lustre: Skipped 3 previous similar messages [ 4795.136506] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 21:44:39 (1778291079) [ 4808.103694] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 21:44:52 (1778291092) [ 4810.848220] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 21:44:54 (1778291094) [ 4813.649541] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 21:44:57 (1778291097) [ 4816.352229] LustreError: 8391:0:(ofd_dev.c:1917:ofd_statfs_hdl()) cfs_fail_timeout id 242 sleeping for 10000ms [ 4816.355600] LustreError: 8391:0:(ofd_dev.c:1917:ofd_statfs_hdl()) Skipped 1 previous similar message [ 4816.768061] LustreError: 42869:0:(ofd_dev.c:1917:ofd_statfs_hdl()) cfs_fail_timeout interrupted [ 4819.126265] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 21:45:03 (1778291103) [ 4821.715198] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 21:45:05 (1778291105) [ 4824.270488] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 21:45:08 (1778291108) [ 4826.839571] Lustre: DEBUG MARKER: == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 21:45:10 (1778291110) [ 4841.694286] Lustre: DEBUG MARKER: == sanity test 119f: dio vs dio race ===================== 21:45:25 (1778291125) [ 4857.817159] Lustre: DEBUG MARKER: == sanity test 119g: dio vs buffered I/O race ============ 21:45:41 (1778291141) [ 4873.611710] Lustre: DEBUG MARKER: == sanity test 119h: basic tests of memory unaligned dio ========================================================== 21:45:57 (1778291157) [ 4884.601479] Lustre: DEBUG MARKER: SKIP: sanity test_119i skipping ALWAYS excluded test 119i [ 4885.275755] Lustre: DEBUG MARKER: == sanity test 119j: basic tests of hybrid IO switching == 21:46:09 (1778291169) [ 4889.277596] Lustre: DEBUG MARKER: == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 21:46:13 (1778291173) [ 4892.128587] Lustre: DEBUG MARKER: == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 21:46:16 (1778291176) [ 4892.794484] Lustre: DEBUG MARKER: SKIP: sanity test_119n need ZFS server without unaligned_dio support [ 4893.519356] Lustre: DEBUG MARKER: == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 21:46:17 (1778291177) [ 4894.123595] Lustre: DEBUG MARKER: SKIP: sanity test_119o need ldiskfs without unaligned_dio support. [ 4894.778833] Lustre: DEBUG MARKER: == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 21:46:18 (1778291178) [ 4897.639477] Lustre: DEBUG MARKER: == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 21:46:21 (1778291181) [ 4902.108311] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 21:46:26 (1778291186) [ 4905.570248] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 21:46:29 (1778291189) [ 4908.975212] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 21:46:32 (1778291192) [ 4912.666105] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 21:46:36 (1778291196) [ 4915.902902] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 21:46:39 (1778291199) [ 4926.210156] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 21:46:50 (1778291210) [ 4936.856689] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 21:47:00 (1778291220) [ 5102.944817] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 21:49:46 (1778291386) [ 5105.683028] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 21:49:49 (1778291389) [ 5109.379539] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 5110.502185] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 5125.271103] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 9 sec [ 5130.880803] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 5190.820416] Lustre: DEBUG MARKER: ls -l 5000 files without statahead: 42 sec [ 5210.529381] Lustre: DEBUG MARKER: ls -l 5000 files with statahead: 16 sec [ 5211.175731] Lustre: DEBUG MARKER: ls -l done [ 5252.789519] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity: 40 seconds [ 5256.285161] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 21:52:20 (1778291540) [ 5259.686656] Lustre: DEBUG MARKER: statx -l 100 files without statahead: 1 sec [ 5260.727710] Lustre: DEBUG MARKER: statx -l 100 files with statahead: 0 sec [ 5273.412361] Lustre: DEBUG MARKER: statx -l 1000 files without statahead: 7 sec [ 5277.829739] Lustre: DEBUG MARKER: statx -l 1000 files with statahead: 3 sec [ 5387.188903] Lustre: DEBUG MARKER: statx -l 10000 files without statahead: 76 sec [ 5425.822669] Lustre: DEBUG MARKER: statx -l 10000 files with statahead: 31 sec [ 5426.472246] Lustre: DEBUG MARKER: statx -l done [ 5510.156692] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ab.sanity: 83 seconds [ 5513.482702] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 21:56:37 (1778291797) [ 5517.087173] Lustre: DEBUG MARKER: statx -c 1 [ 5518.041050] Lustre: DEBUG MARKER: statx -c 1 [ 5528.459429] Lustre: DEBUG MARKER: statx -c 1 [ 5531.633896] Lustre: DEBUG MARKER: statx -c 1 [ 5609.482609] Lustre: DEBUG MARKER: statx -c 1 [ 5633.626775] Lustre: DEBUG MARKER: statx -c 1 [ 5634.131296] Lustre: DEBUG MARKER: statx -c 1 [ 5687.666396] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ac.sanity: 53 seconds [ 5690.584085] Lustre: DEBUG MARKER: statx --cached=always -D 100 files without statahead: 0 sec [ 5691.329689] Lustre: DEBUG MARKER: statx --cached=always -D 100 files with statahead: 0 sec [ 5695.499840] Lustre: DEBUG MARKER: statx --cached=always -D 1000 files without statahead: 0 sec [ 5696.069350] Lustre: DEBUG MARKER: statx --cached=always -D 1000 files with statahead: 0 sec [ 5720.292488] Lustre: DEBUG MARKER: statx --cached=always -D 10000 files without statahead: 0 sec [ 5721.029870] Lustre: DEBUG MARKER: statx --cached=always -D 10000 files with statahead: 1 sec [ 6001.503025] Lustre: DEBUG MARKER: statx --cached=always -D 100000 files without statahead: 2 sec [ 6003.660124] Lustre: DEBUG MARKER: statx --cached=always -D 100000 files with statahead: 1 sec [ 6004.216678] Lustre: DEBUG MARKER: statx --cached=always -D done [ 6811.842547] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ac.sanity: 807 seconds [ 6814.126066] Lustre: DEBUG MARKER: == sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:18:18 (1778293098) [ 6816.829628] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 6817.774946] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 6826.945635] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 6 sec [ 6831.728656] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 3 sec [ 6910.109371] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 55 sec [ 6957.402482] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 41 sec [ 6957.955945] Lustre: DEBUG MARKER: ls -l done [ 7014.141779] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 55 seconds [ 7207.203929] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 1 sec [ 7208.214748] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 1 sec [ 7218.435781] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 6 sec [ 7223.353266] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 4 sec [ 7314.042480] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 66 sec [ 7383.873786] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 58 sec [ 7384.931887] Lustre: DEBUG MARKER: ls -l done [ 7513.262619] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ad.sanity: 126 seconds [ 7955.634795] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 22:37:18 (1778294238) [ 7983.872917] Lustre: DEBUG MARKER: ls done [ 8007.124625] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 22:38:10 (1778294290) [ 8016.294582] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 22:38:19 (1778294299) [ 8033.380203] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 22:38:36 (1778294316) [ 8107.073901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8108.309137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8146.851516] Lustre: 81208:0:(mdt_handler.c:4705:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 8278.984269] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 22:42:40 (1778294560) [ 8459.989568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8459.998943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8872.496265] Lustre: 22440:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 18468 > trans_max 3200 [ 8872.504042] Lustre: 22440:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 8872.509301] Lustre: 22440:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1197/1197/0, xattr_set: 1795/17008/0 [ 8872.516904] Lustre: 22440:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/151/0, punch: 0/0/0, quota 0/0/0 [ 8872.523913] Lustre: 22440:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 8872.529094] Lustre: 22440:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 8872.535280] CPU: 0 PID: 22440 Comm: mdt00_005 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 8872.543318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 8872.550274] Call Trace: [ 8872.553063] ? dump_stack+0xbb/0x10e [ 8872.557256] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 8872.559801] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 8872.564947] ? lod_trans_start+0x109/0x4c0 [lod] [ 8872.568627] ? mdd_trans_start+0x18/0x30 [mdd] [ 8872.570884] ? mdd_unlink+0x758/0x1350 [mdd] [ 8872.573766] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 8872.576249] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 8872.580397] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 8872.584647] ? mdt_reint+0x163/0x190 [mdt] [ 8872.587727] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 8872.591179] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [ 8872.595225] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 8872.599194] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8872.602079] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 8872.606516] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 8872.611082] ? kthread+0x1d1/0x200 [ 8872.612692] ? set_kthread_struct+0x70/0x70 [ 8872.614657] ? ret_from_fork+0x1f/0x30 [ 8933.443933] Lustre: 59182:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30558 > trans_max 3200 [ 8933.451842] Lustre: 59182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 8933.459382] Lustre: 59182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 8933.466909] Lustre: 59182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/151/0, punch: 0/0/0, quota 0/0/0 [ 8933.473120] Lustre: 59182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 8933.484588] Lustre: 59182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 8933.489601] CPU: 2 PID: 59182 Comm: mdt00_008 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 8933.498169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 8933.506025] Call Trace: [ 8933.507127] ? dump_stack+0xbb/0x10e [ 8933.510084] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 8933.513894] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 8933.516697] ? lod_trans_start+0x109/0x4c0 [lod] [ 8933.519091] ? mdd_trans_start+0x18/0x30 [mdd] [ 8933.520997] ? mdd_unlink+0x758/0x1350 [mdd] [ 8933.523072] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 8933.527673] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 8933.531951] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 8933.533922] ? mdt_reint+0x163/0x190 [mdt] [ 8933.539184] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 8933.546298] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [ 8933.552104] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 8933.555542] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8933.557809] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 8933.561391] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 8933.567314] ? kthread+0x1d1/0x200 [ 8933.568921] ? set_kthread_struct+0x70/0x70 [ 8933.570921] ? ret_from_fork+0x1f/0x30 [ 8993.916507] Lustre: 6509:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30587 > trans_max 3200 [ 8993.921766] Lustre: 6509:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 8993.925967] Lustre: 6509:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 8993.930466] Lustre: 6509:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/180/0, punch: 0/0/0, quota 0/0/0 [ 8993.934745] Lustre: 6509:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 8993.940110] Lustre: 6509:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 8993.944673] CPU: 3 PID: 6509 Comm: mdt00_001 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 8993.948960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 8993.952963] Call Trace: [ 8993.954646] ? dump_stack+0xbb/0x10e [ 8993.957672] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 8993.960168] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 8993.962755] ? lod_trans_start+0x109/0x4c0 [lod] [ 8993.965398] ? mdd_trans_start+0x18/0x30 [mdd] [ 8993.967537] ? mdd_unlink+0x758/0x1350 [mdd] [ 8993.969215] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 8993.971194] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 8993.972866] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 8993.974653] ? mdt_reint+0x163/0x190 [mdt] [ 8993.976143] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 8993.978296] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [ 8993.980430] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 8993.983040] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 8993.985048] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 8993.987018] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 8993.989072] ? kthread+0x1d1/0x200 [ 8993.990172] ? set_kthread_struct+0x70/0x70 [ 8993.991647] ? ret_from_fork+0x1f/0x30 [ 9054.908830] Lustre: 22440:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 30558 > trans_max 3200 [ 9054.916520] Lustre: 22440:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [ 9054.920332] Lustre: 22440:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 2003/2003/0, xattr_set: 3004/28292/0 [ 9054.924625] Lustre: 22440:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/151/0, punch: 0/0/0, quota 0/0/0 [ 9054.929482] Lustre: 22440:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [ 9054.938073] Lustre: 22440:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 9054.944068] CPU: 0 PID: 22440 Comm: mdt00_005 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [ 9054.948584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 9054.952352] Call Trace: [ 9054.955124] ? dump_stack+0xbb/0x10e [ 9054.958468] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [ 9054.960542] ? top_trans_start+0x599/0xd80 [ptlrpc] [ 9054.964084] ? lod_trans_start+0x109/0x4c0 [lod] [ 9054.966737] ? mdd_trans_start+0x18/0x30 [mdd] [ 9054.968566] ? mdd_unlink+0x758/0x1350 [mdd] [ 9054.970501] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [ 9054.973010] ? mdt_reint_rec+0x139/0x2b0 [mdt] [ 9054.975085] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [ 9054.976973] ? mdt_reint+0x163/0x190 [mdt] [ 9054.979021] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [ 9054.981664] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [ 9054.988022] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [ 9054.993286] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 9054.996508] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [ 9054.998797] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [ 9055.000655] ? kthread+0x1d1/0x200 [ 9055.001841] ? set_kthread_struct+0x70/0x70 [ 9055.003137] ? ret_from_fork+0x1f/0x30 [ 9077.995254] Lustre: DEBUG MARKER: == sanity test 123g: Test for stat-ahead advise ========== 22:55:58 (1778295358) [ 9129.004386] Lustre: DEBUG MARKER: == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 22:56:52 (1778295412) [10158.158441] Lustre: DEBUG MARKER: == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 23:14:01 (1778296441) [10236.851994] Lustre: DEBUG MARKER: == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 23:15:20 (1778296520) [10239.654870] Lustre: DEBUG MARKER: == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 23:15:23 (1778296523) [10240.190701] Lustre: DEBUG MARKER: SKIP: sanity test_123k mdtest not found [10240.704331] Lustre: DEBUG MARKER: == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 23:15:24 (1778296524) [10311.460909] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 23:16:35 (1778296595) [10312.060620] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [10323.996806] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff93d549846000 [10324.465170] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff93d549846000 [10324.939750] Lustre: DEBUG MARKER: LRU=1015 [10325.415131] Lustre: DEBUG MARKER: LIMIT=46162 [10325.892135] Lustre: DEBUG MARKER: LVF=5457500 [10326.366115] Lustre: DEBUG MARKER: OLD_LVF=100 [10326.841495] Lustre: DEBUG MARKER: Sleep 50 sec [10377.576894] Lustre: DEBUG MARKER: Dropped 430 locks in 50s [10378.041493] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [10383.636601] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 23:17:47 (1778296667) [10404.221714] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [10469.752135] Lustre: DEBUG MARKER: ls -la time: 65 seconds [10470.249884] Lustre: DEBUG MARKER: lru_size = 400 [10512.047211] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [10565.838443] Lustre: DEBUG MARKER: ls -la time: 51 seconds [10566.315535] Lustre: DEBUG MARKER: lru_size = 4006 [10566.799984] Lustre: DEBUG MARKER: ls -la is 21% faster with lru resize enabled [10579.659556] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 23:21:03 (1778296863) [10607.641555] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 23:21:31 (1778296891) [10631.717028] Lustre: DEBUG MARKER: == sanity test 124e: LFRU keep priv locks from eviction == 23:21:55 (1778296915) [10648.520063] Lustre: DEBUG MARKER: == sanity test 124f: LFRU priv threshold inc/dec adjustment ========================================================== 23:22:12 (1778296932) [10678.877308] Lustre: DEBUG MARKER: == sanity test 124g: LFRU performance test =============== 23:22:42 (1778296962) [10852.711396] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 23:25:36 (1778297136) [10854.909399] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 23:25:38 (1778297138) [10856.970932] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 23:25:41 (1778297141) [10859.617825] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 23:25:43 (1778297143) [10862.486839] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [10871.730962] Lustre: DEBUG MARKER: == sanity test 127d: OSC RPC latency histograms for read and write latency ========================================================== 23:25:55 (1778297155) [10875.210674] Lustre: DEBUG MARKER: == sanity test 127e: client IO latency histograms by size ========================================================== 23:25:59 (1778297159) [10884.755595] Lustre: DEBUG MARKER: == sanity test 127f: OST IO latency histograms by size === 23:26:08 (1778297168) [10897.049914] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 23:26:21 (1778297181) [10899.329383] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 23:26:23 (1778297183) [10906.143262] Lustre: 6509:0:(osd_handler.c:613:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513272, FID: [0x20000040b:0x904c:0x0]) is approaching max size limit [10910.149791] Lustre: 6508:0:(osd_handler.c:609:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513272, FID: [0x20000040b:0x904c:0x0]) has reached max size limit [10922.909260] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 23:26:46 (1778297206) [10925.557022] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 23:26:49 (1778297209) [10928.893196] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 23:26:52 (1778297212) [10932.600805] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 23:26:56 (1778297216) [10933.316912] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [10934.035179] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 23:26:57 (1778297217) [10958.044849] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 23:27:21 (1778297241) [10961.230643] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 23:27:25 (1778297245) [10975.118684] Lustre: 11281:0:(osd_handler.c:2191:osd_trans_start()) lustre-MDT0000: credits 6558 > trans_max 3200 [10975.121642] Lustre: 11281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 4/16/0, destroy: 1/4/0 [10975.123910] Lustre: 11281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 403/403/0, xattr_set: 604/5892/0 [10975.126684] Lustre: 11281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 20/151/0, punch: 0/0/0, quota 0/0/0 [10975.129633] Lustre: 11281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 5/84/0, delete: 2/5/0 [10975.132316] Lustre: 11281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [10975.134966] CPU: 0 PID: 11281 Comm: mdt00_004 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [10975.138045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [10975.140384] Call Trace: [10975.141026] ? dump_stack+0xbb/0x10e [10975.142028] ? osd_trans_start+0x8db/0x9d0 [osd_ldiskfs] [10975.143508] ? top_trans_start+0x599/0xd80 [ptlrpc] [10975.145000] ? lod_trans_start+0x109/0x4c0 [lod] [10975.146355] ? mdd_trans_start+0x18/0x30 [mdd] [10975.147532] ? mdd_unlink+0x758/0x1350 [mdd] [10975.148644] ? mdt_reint_unlink+0x14aa/0x1a30 [mdt] [10975.149899] ? mdt_reint_rec+0x139/0x2b0 [mdt] [10975.151068] ? mdt_reint_internal+0x6a0/0xdc0 [mdt] [10975.152342] ? mdt_reint+0x163/0x190 [mdt] [10975.153384] ? tgt_handle_request0+0x137/0xaf0 [ptlrpc] [10975.154801] ? tgt_request_handle+0x575/0x1e70 [ptlrpc] [10975.156245] ? ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [10975.157796] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [10975.159184] ? ptlrpc_main+0xce8/0x1400 [ptlrpc] [10975.160412] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [10975.161704] ? kthread+0x1d1/0x200 [10975.162523] ? set_kthread_struct+0x70/0x70 [10975.163488] ? ret_from_fork+0x1f/0x30 [10977.181924] Lustre: DEBUG MARKER: == sanity test 130h: FIEMAP deadlock ===================== 23:27:41 (1778297261) [10986.258243] Lustre: DEBUG MARKER: == sanity test 130i: FIEMAP (DoM file) =================== 23:27:50 (1778297270) [11007.347572] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 23:28:11 (1778297291) [11010.598251] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 23:28:14 (1778297294) [11013.935209] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 23:28:17 (1778297297) [11016.734309] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 23:28:20 (1778297300) [11020.060386] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 23:28:23 (1778297303) [11023.469547] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 23:28:27 (1778297307) [11034.271406] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 23:28:37 (1778297317) [11040.198249] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 23:28:43 (1778297323) [11062.799518] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 23:29:06 (1778297346) [11074.137502] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 23:29:17 (1778297357) [11080.141579] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 23:29:23 (1778297363) [11088.867133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11088.870585] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11088.872888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11088.881876] Lustre: Skipped 3 previous similar messages [11088.888612] Lustre: Skipped 3 previous similar messages [11093.743657] Lustre: server umount lustre-MDT0000 complete [11093.988103] LustreError: 59181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.000386] LustreError: 59181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11097.853349] LustreError: 36205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778297381 with bad export cookie 8888496214462641458 [11097.859978] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11097.860676] LustreError: 36205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11098.148958] Lustre: server umount lustre-MDT0001 complete [11111.367814] Lustre: server umount lustre-OST0000 complete [11123.362385] Lustre: server umount lustre-OST0001 complete [11128.376582] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11129.482283] Key type lgssc unregistered [11129.616489] LNet: 103542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11129.618840] LNetError: 103542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11129.627403] LNet: Removed LNI 192.168.204.106@tcp [11130.037130] Key type .llcrypt unregistered [11130.038768] Key type ._llcrypt unregistered [11137.412569] Key type ._llcrypt registered [11137.413626] Key type .llcrypt registered [11137.451510] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11137.795920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11137.835113] alg: No test for adler32 (adler32-zlib) [11138.686891] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [11138.765971] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11140.351215] Key type lgssc registered [11140.894233] Lustre: Echo OBD driver; http://www.lustre.org/ [11146.492699] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11151.909658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11151.919082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.090723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11154.398941] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11158.557840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.708175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11160.119521] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11161.389850] Lustre: 106822:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11164.704284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11164.885659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11167.218077] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11168.999430] LustreError: 107328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.003068] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:45450 to 0x280000400:45473) [11169.005577] LustreError: 107328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11171.499654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11171.645087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11173.657073] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11175.228547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:45281 to 0x2c0000400:45313) [11177.281219] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:57020 to 0x2c0000403:57089) [11177.283499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:57410 to 0x280000403:57473) [11182.480552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11183.881848] Lustre: 108981:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11192.713674] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 23:31:16 (1778297476) [11195.721334] LNet: 109469:0:(debug.c:370:cfs_str2mask()) unknown mask ''. [11195.721334] mask usage: [+|-] ... [11196.084112] Lustre: DEBUG MARKER:  [11196.085671] Lustre: DEBUG MARKER:  [11196.273391] LNet: 109547:0:(debug.c:370:cfs_str2mask()) unknown mask ''. [11196.273391] mask usage: [+|-] ... [11196.278919] LNet: 109547:0:(debug.c:370:cfs_str2mask()) Skipped 4 previous similar messages [11214.620299] LNet: 111131:0:(debug.c:370:cfs_str2mask()) unknown mask ''. [11214.620299] mask usage: [+|-] ... [11214.624668] LNet: 111131:0:(debug.c:370:cfs_str2mask()) Skipped 2 previous similar messages [11214.926595] Lustre: DEBUG MARKER:  [11214.927653] Lustre: DEBUG MARKER:  [11233.765136] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.770451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.774277] Lustre: Skipped 2 previous similar messages [11233.778181] Lustre: Skipped 1 previous similar message [11238.158009] Lustre: server umount lustre-MDT0000 complete [11238.883612] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.894421] LustreError: 105557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11242.150165] LustreError: 105537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778297526 with bad export cookie 1008834762982171081 [11242.152873] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11242.155280] LustreError: 105537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11242.275007] Lustre: server umount lustre-MDT0001 complete [11255.314479] Lustre: server umount lustre-OST0000 complete [11268.355362] Lustre: server umount lustre-OST0001 complete [11276.641516] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11278.287339] Key type lgssc unregistered [11278.483591] LNet: 114885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.489626] LNetError: 114885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.502697] LNet: Removed LNI 192.168.204.106@tcp [11279.041153] Key type .llcrypt unregistered [11279.043036] Key type ._llcrypt unregistered [11286.055641] Key type ._llcrypt registered [11286.057622] Key type .llcrypt registered [11286.130905] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11286.496546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11286.522795] alg: No test for adler32 (adler32-zlib) [11287.418701] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [11287.534296] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11289.135221] Key type lgssc registered [11289.794108] Lustre: Echo OBD driver; http://www.lustre.org/ [11297.188261] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11303.337623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11303.347136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.553382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11306.186224] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11311.317666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.537922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11313.142084] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11314.565122] Lustre: 118166:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11318.880453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11319.086065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11321.642085] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11322.153030] LustreError: 118675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.159546] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:45450 to 0x280000400:45505) [11322.165267] LustreError: 118675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11327.303087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11327.464533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11329.515561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:57410 to 0x280000403:57505) [11329.516075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:57020 to 0x2c0000403:57121) [11329.571044] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:45281 to 0x2c0000400:45345) [11329.866014] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [11334.959490] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11336.653819] Lustre: 120321:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11344.221507] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 23:33:48 (1778297628) [11766.962173] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 23:40:51 (1778298051) [11770.786587] Lustre: *** cfs_fail_loc=327, val=500*** [11787.063195] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 23:41:11 (1778298071) [11789.748021] Lustre: *** cfs_fail_loc=328, val=500*** [11789.749604] Lustre: Skipped 1284 previous similar messages [11790.312901] Lustre: 125469:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a08c4f34a80 x1864680148589568/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:530/0 lens 648/0 e 0 to 0 dl 1778298085 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11791.328819] Lustre: *** cfs_fail_loc=328, val=500*** [11791.330976] Lustre: Skipped 252 previous similar messages [11791.333094] Lustre: 120534:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a09f727c700 x1864680148590336/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:531/0 lens 648/0 e 0 to 0 dl 1778298086 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11792.354784] Lustre: 116899:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a09f727df80 x1864680148591104/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:532/0 lens 648/0 e 0 to 0 dl 1778298087 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11793.377272] Lustre: *** cfs_fail_loc=328, val=500*** [11793.378476] Lustre: Skipped 1 previous similar message [11794.401257] Lustre: 116897:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a09d09a4700 x1864680148593024/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:534/0 lens 648/0 e 0 to 0 dl 1778298089 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11794.410046] Lustre: 116897:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 1 previous similar message [11797.793357] Lustre: *** cfs_fail_loc=328, val=500*** [11797.794649] Lustre: Skipped 125 previous similar messages [11798.817382] Lustre: 120534:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a08cb8b1c00 x1864680148659072/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:538/0 lens 648/0 e 0 to 0 dl 1778298093 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11798.826398] Lustre: 120534:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 3 previous similar messages [11806.113670] Lustre: *** cfs_fail_loc=328, val=500*** [11806.115641] Lustre: Skipped 66 previous similar messages [11807.201365] Lustre: 125469:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) @@@ Too many granted locks, reject current enqueue request and let the client retry later req@ffff8a08d8f67800 x1864680148709632/t0(0) o101->069a1fd9-c543-4849-ab73-30261154c280@192.168.204.6@tcp:547/0 lens 648/0 e 0 to 0 dl 1778298102 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [11807.209503] Lustre: 125469:0:(ldlm_lockd.c:1326:ldlm_handle_enqueue()) Skipped 7 previous similar messages [11810.072681] LustreError: 183283:0:(ldlm_lockd.c:3240:lock_reclaim_threshold_mb_store()) Failed to set lock_reclaim_threshold_mb, rc = -22. [11815.096726] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [11815.617923] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [11816.175849] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 23:41:40 (1778298100) [11829.136980] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 23:41:53 (1778298113) [11859.367353] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 23:42:23 (1778298143) [11873.837615] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 23:42:37 (1778298157) [11889.237971] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 23:42:53 (1778298173) [11900.389662] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 23:43:04 (1778298184) [11915.530591] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 23:43:19 (1778298199) [11933.649757] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 23:43:37 (1778298217) [11946.537702] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 23:43:50 (1778298230) [11961.208207] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 23:44:05 (1778298245) [11963.833722] Lustre: DEBUG MARKER: == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 23:44:07 (1778298247) [11977.361918] Lustre: DEBUG MARKER: == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 23:44:21 (1778298261) [11993.347235] Lustre: DEBUG MARKER: == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 23:44:37 (1778298277) [11994.120100] Lustre: DEBUG MARKER: SKIP: sanity test_150ic only check on DoM component [11994.984931] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 23:44:38 (1778298278) [12002.600432] bash (193563): drop_caches: 1 [12007.969708] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 23:44:51 (1778298291) [12010.676448] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 23:44:54 (1778298294) [12012.996055] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 23:44:57 (1778298297) [12015.304098] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 23:44:59 (1778298299) [12017.471343] Lustre: DEBUG MARKER: == sanity test 154C: lfs fid2path on OST FID ============= 23:45:01 (1778298301) [12033.152674] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 23:45:17 (1778298317) [12033.755452] LustreError: 116908:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [12036.638805] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 23:45:20 (1778298320) [12037.076300] LustreError: 117790:0:(fld_handler.c:268:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [12037.079401] LustreError: 117790:0:(fld_handler.c:268:fld_server_lookup()) Skipped 3 previous similar messages [12039.835422] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 23:45:23 (1778298323) [12042.277476] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 23:45:26 (1778298326) [12045.211505] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 23:45:29 (1778298329) [12047.600964] Lustre: DEBUG MARKER: == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 23:45:31 (1778298331) [12066.727568] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 23:45:50 (1778298350) [12069.124045] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 23:45:53 (1778298353) [12411.811425] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 23:51:35 (1778298695) [12414.124946] Lustre: DEBUG MARKER: == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 23:51:38 (1778298698) [12426.196181] Lustre: DEBUG MARKER: == sanity test 154j: fid2path for long path crossing MDT boundary ========================================================== 23:51:50 (1778298710) [12433.833403] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 23:51:57 (1778298717) [12438.356359] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 23:52:02 (1778298722) [12442.891641] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 23:52:06 (1778298726) [12447.452472] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 23:52:11 (1778298731) [12451.970526] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 23:52:15 (1778298735) [12469.688024] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 23:52:33 (1778298753) [12495.792656] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 23:52:59 (1778298779) [12521.939965] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 23:53:25 (1778298805) [12546.240827] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 23:53:50 (1778298830) [12549.112068] Lustre: DEBUG MARKER: Turn on read and write cache [12550.363187] Lustre: DEBUG MARKER: Write data and read it back. [12550.937377] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [12552.276096] Lustre: DEBUG MARKER: cache hits: before: 28718, after: 28721 [12552.842489] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12553.900061] Lustre: DEBUG MARKER: cache hits:: before: 28721, after: 28724 [12554.470423] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [12555.707459] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12556.995170] Lustre: DEBUG MARKER: cache hits:: before: 28724, after: 28727 [12557.546676] Lustre: DEBUG MARKER: Write data and read it back. [12558.080785] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [12559.471805] Lustre: DEBUG MARKER: cache hits:: before: 28727, after: 28730 [12560.079135] Lustre: DEBUG MARKER: Turn off read and write cache [12561.398213] Lustre: DEBUG MARKER: Write data and read it back [12561.958026] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [12563.247299] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28730 [12563.765406] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [12565.002817] Lustre: DEBUG MARKER: Write data and read it back [12565.514307] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [12566.792169] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28730 [12567.332655] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [12568.602734] Lustre: DEBUG MARKER: cache hits:: before: 28730, after: 28733 [12571.487579] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 23:54:15 (1778298855) [12572.456193] Lustre: lustre-MDD0000: changelog on [12573.326137] Lustre: lustre-MDD0001: changelog on [12578.394805] Lustre: Failing over lustre-MDT0000 [12578.591892] Lustre: server umount lustre-MDT0000 complete [12578.784910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12578.788547] LustreError: 116897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12578.790296] Lustre: Skipped 2 previous similar messages [12578.795669] LustreError: 116897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12581.319831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.350743] LustreError: 119080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.364581] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12581.475947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12581.490092] Lustre: lustre-MDD0000: changelog on [12581.499891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12582.716141] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [12586.464050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12586.983841] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [12587.138265] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [12587.155352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:57960 to 0x2c0000403:57985) [12587.155360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:58347 to 0x280000403:58369) [12587.411400] Lustre: lustre-MDD0000: changelog off [12587.977347] Lustre: lustre-MDD0001: changelog off [12592.270219] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 23:54:36 (1778298876) [12593.235280] Lustre: lustre-MDD0000: changelog on [12596.894739] Lustre: lustre-MDD0001: changelog off [12598.649101] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 23:54:42 (1778298882) [12599.778308] Lustre: lustre-MDD0000: changelog on [12599.781051] Lustre: Skipped 1 previous similar message [12603.954512] Lustre: lustre-MDD0001: changelog off [12603.955866] Lustre: Skipped 1 previous similar message [12605.613446] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 23:54:49 (1778298889) [12610.164447] Lustre: lustre-MDD0001: changelog off [12610.167120] Lustre: Skipped 1 previous similar message [12611.975984] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 23:54:56 (1778298896) [12612.905052] Lustre: lustre-MDD0000: changelog on [12612.906652] Lustre: Skipped 3 previous similar messages [12618.582282] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 23:55:02 (1778298902) [12623.343917] Lustre: DEBUG MARKER: 1778298907: creating first dirs [12640.191455] Lustre: *** cfs_fail_loc=1313, val=3*** [12640.194227] Lustre: 125469:0:(mdd_dir.c:971:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [12640.199986] Lustre: 125469:0:(mdd_dir.c:1054:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [12640.208337] Lustre: 214131:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 19s with 4 unprocessed records [12647.017256] Lustre: lustre-MDD0001: changelog off [12647.019008] Lustre: Skipped 3 previous similar messages [12648.664898] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 23:55:32 (1778298932) [12649.658947] Lustre: lustre-MDD0000: changelog on [12649.660156] Lustre: Skipped 3 previous similar messages [12656.662962] Lustre: 116897:0:(mdd_dir.c:1054:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [12656.665504] Lustre: 116897:0:(mdd_dir.c:1054:mdd_changelog_store()) Skipped 1 previous similar message [12656.674087] Lustre: 216589:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 5s with 4 unprocessed records [12656.678889] Lustre: 216589:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [12663.328270] Lustre: lustre-MDD0001: changelog off [12663.329549] Lustre: Skipped 1 previous similar message [12665.290344] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 23:55:49 (1778298949) [12685.459854] Lustre: *** cfs_fail_loc=1316, val=0*** [12685.464345] Lustre: 116898:0:(mdd_dir.c:1054:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [12685.474522] Lustre: 116898:0:(mdd_dir.c:1054:mdd_changelog_store()) Skipped 1 previous similar message [12685.486557] Lustre: 219042:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 17s with 3 unprocessed records [12685.495719] Lustre: 219042:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [12686.525274] Lustre: Failing over lustre-MDT0000 [12686.580969] Lustre: Failing over lustre-MDT0001 [12688.869194] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12689.382120] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.385305] Lustre: Skipped 1 previous similar message [12689.387884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12689.389400] Lustre: Skipped 1 previous similar message [12689.773128] Lustre: server umount lustre-MDT0001 complete [12689.840260] Lustre: server umount lustre-MDT0000 complete [12693.299708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.350499] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12693.472573] LustreError: 118684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12693.477449] LustreError: 118684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12693.527161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12693.545950] Lustre: lustre-MDD0000: changelog on [12693.547059] Lustre: Skipped 3 previous similar messages [12693.548554] Lustre: 220220:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 32 to index 33, being cleared now [12693.556098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [12693.986722] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [12695.035027] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [12697.850202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12697.955700] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12697.958671] Lustre: Skipped 3 previous similar messages [12697.960402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [12698.016373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12698.030526] Lustre: 220954:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 14 to index 15, being cleared now [12698.048778] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [12699.102884] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [12699.207708] Lustre: lustre-MDT0001: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [12699.218760] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [12699.225857] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:45349 to 0x2c0000400:45377) [12699.225882] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:45510 to 0x280000400:45537) [12703.210272] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12703.212107] Lustre: Skipped 1 previous similar message [12703.302379] Lustre: lustre-MDT0000: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. [12703.324293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:58371 to 0x280000403:58401) [12703.324401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:57960 to 0x2c0000403:58017) [12708.707543] Lustre: lustre-MDD0001: changelog off [12708.709515] Lustre: Skipped 1 previous similar message [12710.472473] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 23:56:34 (1778298994) [12713.837165] LustreError: 222902:0:(mdd_device.c:2063:mdd_changelog_user_purge()) cfs_race id 1315 sleeping [12716.175876] LustreError: 223035:0:(mdd_device.c:1778:mdd_changelog_user_register()) cfs_fail_race id 1315 waking [12716.181749] LustreError: 222902:0:(mdd_device.c:2063:mdd_changelog_user_purge()) cfs_fail_race id 1315 awake: rc=2658 [12717.078681] LustreError: 223226:0:(mdd_device.c:2063:mdd_changelog_user_purge()) cfs_fail_race id 1315 waking [12724.475799] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 23:56:48 (1778299008) [12731.982920] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 23:56:56 (1778299016) [12734.637362] LustreError: 220334:0:(mdd_dir.c:1026:mdd_changelog_store()) cfs_fail_timeout id 15d sleeping for 3000ms [12737.655177] LustreError: 220334:0:(mdd_dir.c:1026:mdd_changelog_store()) cfs_fail_timeout id 15d awake [12745.697525] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 23:57:09 (1778299029) [12754.410713] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 23:57:18 (1778299038) [12759.237585] LustreError: 220234:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_race id 15f sleeping [12761.242738] LustreError: 220232:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_fail_race id 15f waking [12761.246058] LustreError: 220232:0:(mdd_device.c:398:llog_changelog_cancel_cb()) Skipped 1 previous similar message [12761.249416] LustreError: 220234:0:(mdd_device.c:398:llog_changelog_cancel_cb()) cfs_fail_race id 15f awake: rc=2990 [12766.906644] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 23:57:30 (1778299050) [12768.056353] Lustre: lustre-MDD0000: changelog on [12768.059226] Lustre: Skipped 13 previous similar messages [13462.320175] LustreError: 228934:0:(mdd_device.c:412:llog_changelog_cancel_cb()) cfs_race id 16c sleeping [13464.326155] LustreError: 220234:0:(llog_osd.c:1130:llog_osd_next_block()) cfs_fail_race id 16c waking [13464.329793] LustreError: 228934:0:(mdd_device.c:412:llog_changelog_cancel_cb()) cfs_fail_race id 16c awake: rc=2993 [13469.671832] Lustre: lustre-MDD0001: changelog off [13469.674063] Lustre: Skipped 13 previous similar messages [13471.364840] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 00:09:15 (1778299755) [13472.337578] Lustre: lustre-MDD0000: changelog on [13472.338817] Lustre: Skipped 1 previous similar message [13473.533295] LustreError: 230914:0:(mdd_device.c:1732:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [13473.821373] Lustre: 230962:0:(mdd_device.c:1749:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [13474.099145] LustreError: 231010:0:(mdd_device.c:1741:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [13481.979969] Lustre: lustre-MDD0000: changelog off [13481.981749] Lustre: Skipped 1 previous similar message [13484.291313] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 00:09:28 (1778299768) [13487.391304] Lustre: Failing over lustre-MDT0000 [13487.585372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13487.588132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13487.588572] Lustre: Skipped 4 previous similar messages [13487.590912] Lustre: Skipped 3 previous similar messages [13487.690205] Lustre: server umount lustre-MDT0000 complete [13490.738061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13490.783063] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13490.903856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13490.937197] Lustre: 233338:0:(mdd_device.c:626:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90156 to index 18446744073709551615, being cleared now [13490.948670] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13492.153839] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13492.205225] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13496.294725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13496.296528] Lustre: Skipped 2 previous similar messages [13496.310220] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [13496.325799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000403:58019 to 0x2c0000403:58049) [13496.325804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000403:58403 to 0x280000403:58433) [13498.398431] Lustre: lustre-MDD0001: changelog off [13498.399821] Lustre: Skipped 1 previous similar message [13499.711328] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 00:09:43 (1778299783) [13502.947458] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records * time ========================================================== 00:09:47 (1778299787) [13511.017277] Lustre: 228934:0:(mdd_dir.c:1054:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [13511.019817] Lustre: 228934:0:(mdd_dir.c:1054:mdd_changelog_store()) Skipped 1 previous similar message [13511.027613] Lustre: 235325:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864006s with 500000004 unprocessed records [13516.898188] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 00:10:00 (1778299800) [13523.492044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000403 to 0x280000404 [13523.682211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000403 to 0x2c0000404 [13536.427002] Lustre: *** cfs_fail_loc=18c, val=1210732*** [13536.429914] Lustre: Skipped 1 previous similar message [13536.432622] Lustre: 220233:0:(mdd_dir.c:1054:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [13536.438050] Lustre: 220233:0:(mdd_dir.c:1054:mdd_changelog_store()) Skipped 1 previous similar message [13536.444190] Lustre: 237298:0:(mdd_dir.c:942:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog size 1MB with 1MB space limit [13536.450639] Lustre: 237298:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl3-user1 idle for 19s with 7506 unprocessed records [13536.457239] Lustre: 237298:0:(mdd_trans.c:163:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [13542.775232] Lustre: lustre-MDD0000: changelog off [13542.777840] Lustre: Skipped 3 previous similar messages [13551.705576] Lustre: DEBUG MARKER: == sanity test 160u: changelog rename record type name and sname strings are correct ========================================================== 00:10:35 (1778299835) [13564.788875] Lustre: DEBUG MARKER: == sanity test 160v: setattr preserves mtime update despite inter-client clock skew ========================================================== 00:10:48 (1778299848) [13565.258906] Lustre: DEBUG MARKER: SKIP: sanity test_160v need >= 2 clients [13565.928361] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 00:10:49 (1778299849) [13580.204574] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 00:11:04 (1778299864) [13593.771329] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 00:11:17 (1778299877) [13604.662282] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 00:11:28 (1778299888) [13612.394368] Lustre: lustre-MDD0001: changelog off [13612.395471] Lustre: Skipped 5 previous similar messages [13614.000646] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 00:11:38 (1778299898) [13616.543295] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 00:11:40 (1778299900) [13619.166859] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 00:11:43 (1778299903) [13657.524929] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 00:12:21 (1778299941) [13663.677096] Lustre: Failing over lustre-OST0000 [13663.810414] Lustre: server umount lustre-OST0000 complete [13665.252475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13665.253262] LustreError: 118675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13665.255848] Lustre: Skipped 2 previous similar messages [13665.260025] LustreError: 118675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13666.278093] LustreError: 196779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.282258] LustreError: 196779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13669.349790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13669.525541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13669.532896] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13671.082298] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13671.223205] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13671.223540] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13671.225985] Lustre: Skipped 3 previous similar messages [13671.359554] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13672.486949] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 00:12:36 (1778299956) [13697.474268] Lustre: Failing over lustre-OST0000 [13697.524822] Lustre: server umount lustre-OST0000 complete [13700.582624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13700.595366] LustreError: 118674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13700.604659] LustreError: 118674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13701.309683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13701.446644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13701.459061] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13702.118619] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13703.223843] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13703.224130] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13703.227220] Lustre: Skipped 1 previous similar message [13703.332334] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13704.908666] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 00:13:08 (1778299988) [13719.568267] Lustre: Failing over lustre-OST0000 [13721.073252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13721.073798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13721.075966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13721.075971] Lustre: Skipped 2 previous similar messages [13721.078034] LustreError: Skipped 6 previous similar messages [13721.085074] Lustre: Skipped 2 previous similar messages [13721.664767] Lustre: server umount lustre-OST0000 complete [13722.084368] LustreError: 196778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13722.091723] LustreError: 196778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13725.543708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13725.712940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13725.720977] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13727.099727] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13727.162966] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13727.163761] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13727.167336] Lustre: Skipped 1 previous similar message [13728.067567] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13729.530383] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 00:13:33 (1778300013) [13756.443221] Lustre: Failing over lustre-OST0000 [13756.515266] Lustre: server umount lustre-OST0000 complete [13757.922335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13757.922709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13757.927160] LustreError: Skipped 10 previous similar messages [13757.936763] LustreError: 196780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.951644] LustreError: 196780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13760.592290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13760.726095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13760.734289] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13762.357609] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13762.804726] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13762.805425] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13762.806577] Lustre: Skipped 1 previous similar message [13763.262794] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13765.068744] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 00:14:08 (1778300048) [13778.536775] Lustre: Failing over lustre-OST0000 [13778.587672] Lustre: server umount lustre-OST0000 complete [13778.925030] LustreError: 182845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13778.936455] LustreError: 182845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13781.476262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13781.479551] Lustre: Skipped 1 previous similar message [13782.480501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.590766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13782.597372] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13784.031405] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13784.308525] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13784.308736] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13784.310795] Lustre: Skipped 1 previous similar message [13784.471343] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13786.376236] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 00:14:29 (1778300069) [13792.294947] Lustre: Failing over lustre-OST0000 [13792.739967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13792.741976] Lustre: Skipped 2 previous similar messages [13794.384730] Lustre: server umount lustre-OST0000 complete [13797.860656] LustreError: 118676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.871788] LustreError: 118676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13802.073179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13802.176191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13802.182651] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13803.634199] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13803.895574] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [13803.895871] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13803.897469] Lustre: Skipped 1 previous similar message [13804.440780] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13806.402583] Lustre: DEBUG MARKER: == sanity test 165g: ofd_access_log_reader --keepalive works ========================================================== 00:14:50 (1778300090) [13848.116330] Lustre: Failing over lustre-OST0000 [13848.546961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13848.550384] Lustre: Skipped 3 previous similar messages [13848.552286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13848.554502] Lustre: Skipped 2 previous similar messages [13850.178098] Lustre: server umount lustre-OST0000 complete [13850.607877] LustreError: 118676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13850.612339] LustreError: 118676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13857.503158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13857.639614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13857.647939] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [13858.799992] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [13858.943966] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [13858.944650] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [13858.948123] Lustre: Skipped 1 previous similar message [13859.861629] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug all all [13861.163235] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 00:15:45 (1778300145) [13861.789304] Lustre: DEBUG MARKER: creating a 10 Mb file [13875.084361] Lustre: DEBUG MARKER: starting reads [13875.565572] Lustre: DEBUG MARKER: truncating the file [13876.168823] Lustre: DEBUG MARKER: killing dd [13876.628857] Lustre: DEBUG MARKER: removing the temporary file [13878.560357] Lustre: DEBUG MARKER: == sanity test 170a: test lctl df to handle corrupted log ========================================================== 00:16:02 (1778300162) [13880.874323] Lustre: DEBUG MARKER: == sanity test 170b: check filename encoding ============= 00:16:04 (1778300164) [13889.090468] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 00:16:13 (1778300173) [13894.092820] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 00:16:18 (1778300178) [13897.003104] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 00:16:21 (1778300181) [13897.474291] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [13897.989755] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 00:16:22 (1778300182) [13898.919211] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_module obdecho/obdecho [13904.167822] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 00:16:28 (1778300188) [13905.168438] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_module obdecho/obdecho [13905.219018] Lustre: Echo OBD driver; http://www.lustre.org/ [13911.610931] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 00:16:35 (1778300195) [13931.248901] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 00:16:55 (1778300215) [13935.807544] ODEBUG: object 000000000cce6cac is on stack 00000000a5477fed, but NOT annotated. [13935.811796] WARNING: CPU: 2 PID: 220232 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x15f [13935.816393] Modules linked in: dm_flakey lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix serio_raw ghash_clmulni_intel libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: obdecho] [13935.839332] CPU: 2 PID: 220232 Comm: mdt00_000 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [13935.844571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [13935.848943] RIP: 0010:__debug_object_init.cold.5+0x35/0x15f [13935.851330] Code: 7e a1 48 83 05 23 61 0c 03 01 89 05 59 69 0c 03 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 93 68 99 ff 48 83 05 1b 61 0c 03 01 <0f> 0b 48 83 05 19 61 0c 03 01 48 83 05 19 61 0c 03 01 e9 3f ee ff [13935.860535] RSP: 0018:ffffaa93891774a0 EFLAGS: 00010002 [13935.862220] RAX: 0000000000000050 RBX: ffffaa93891775a8 RCX: 0000000000000000 [13935.864483] RDX: 0000000000000000 RSI: ffff8a0a0211e5a8 RDI: ffff8a0a0211e5a8 [13935.866445] RBP: ffffffffa1f06ae0 R08: 0000000000000000 R09: c0000000ffff7fff [13935.868420] R10: 0000000000000001 R11: ffffaa9389177298 R12: ffffffffa3704088 [13935.870511] R13: 00000000000168a0 R14: ffffffffa3704080 R15: ffff8a08ecfd1d20 [13935.872692] FS: 0000000000000000(0000) GS:ffff8a0a02100000(0000) knlGS:0000000000000000 [13935.875327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13935.877058] CR2: 00007ff7aaac43e0 CR3: 00000000a1016003 CR4: 0000000000170ee0 [13935.879213] Call Trace: [13935.879912] ? show_regs.cold.9+0x22/0x2f [13935.880818] ? __warn+0xc8/0x150 [13935.881523] ? __debug_object_init.cold.5+0x35/0x15f [13935.882564] ? report_bug+0x113/0x140 [13935.883359] ? do_error_trap+0xb6/0x130 [13935.884196] ? do_invalid_op+0x46/0x60 [13935.885296] ? __debug_object_init.cold.5+0x35/0x15f [13935.886541] ? invalid_op+0x14/0x20 [13935.887479] ? __debug_object_init.cold.5+0x35/0x15f [13935.888641] ? lod_set_pool+0x260/0x260 [lod] [13935.890101] debug_object_init+0x22/0x30 [13935.891217] init_timer_key+0x28/0x120 [13935.892062] lod_ost_alloc_qos+0x790/0x1c60 [lod] [13935.893339] ? slab_post_alloc_hook+0x66/0x380 [13935.894486] ? lod_qos_prep_create+0x390/0x1bc0 [lod] [13935.897368] ? __kmalloc+0x1b4/0x4a0 [13935.899642] lod_qos_prep_create+0x134e/0x1bc0 [lod] [13935.902225] lod_prepare_create+0x204/0x460 [lod] [13935.904634] ? osd_declare_create+0x4a2/0x7a0 [osd_ldiskfs] [13935.907840] lod_declare_striped_create+0x270/0xf80 [lod] [13935.910248] ? lod_sub_declare_create+0x111/0x320 [lod] [13935.912538] lod_declare_create+0x3d4/0x9c0 [lod] [13935.914695] ? osd_xattr_get+0x274/0x910 [osd_ldiskfs] [13935.916848] mdd_declare_create_object_internal+0x107/0x4a0 [mdd] [13935.920162] ? lod_alloc_comp_entries+0x290/0x640 [lod] [13935.922936] mdd_declare_create_object.isra.25+0x55/0xc40 [mdd] [13935.926203] mdd_declare_create+0x6a/0x6c0 [mdd] [13935.928008] mdd_create+0x5bd/0x1d00 [mdd] [13935.929658] ? mdt_version_save+0xa8/0x210 [mdt] [13935.932698] mdt_reint_open+0x337c/0x3c10 [mdt] [13935.935565] ? old_init_ucred_common+0x19e/0x850 [mdt] [13935.937718] mdt_reint_rec+0x139/0x2b0 [mdt] [13935.939350] mdt_reint_internal+0x6a0/0xdc0 [mdt] [13935.941041] mdt_intent_open+0x180/0x5b0 [mdt] [13935.942490] mdt_intent_opc.constprop.43+0x153/0xfb0 [mdt] [13935.944562] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] [13935.945880] mdt_intent_policy+0x14b/0x670 [mdt] [13935.947354] ldlm_lock_enqueue+0x43c/0xcd0 [ptlrpc] [13935.948854] ? _raw_read_unlock+0x12/0x30 [13935.950118] ? cfs_hash_rw_unlock+0x11/0x30 [obdclass] [13935.951651] ldlm_handle_enqueue+0xcaf/0x2280 [ptlrpc] [13935.953213] tgt_enqueue+0xd0/0x300 [ptlrpc] [13935.954936] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [13935.956532] tgt_request_handle+0x575/0x1e70 [ptlrpc] [13935.958108] ? obd_export_timed_fini+0xc4/0xe0 [obdclass] [13935.960269] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [13935.962600] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13935.964393] ptlrpc_main+0xce8/0x1400 [ptlrpc] [13935.965906] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [13935.967505] kthread+0x1d1/0x200 [13935.968551] ? set_kthread_struct+0x70/0x70 [13935.970136] ret_from_fork+0x1f/0x30 [13935.971186] ---[ end trace 8ff8e2d705defe2f ]--- [13935.991224] ODEBUG: object 00000000529ae7fb is on stack 00000000af7a18bc, but NOT annotated. [13935.995658] WARNING: CPU: 2 PID: 228934 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x15f [13935.999839] Modules linked in: dm_flakey lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix serio_raw ghash_clmulni_intel libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: obdecho] [13936.015861] CPU: 2 PID: 228934 Comm: mdt00_006 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [13936.020158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [13936.023106] RIP: 0010:__debug_object_init.cold.5+0x35/0x15f [13936.024712] Code: 7e a1 48 83 05 23 61 0c 03 01 89 05 59 69 0c 03 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 93 68 99 ff 48 83 05 1b 61 0c 03 01 <0f> 0b 48 83 05 19 61 0c 03 01 48 83 05 19 61 0c 03 01 e9 3f ee ff [13936.030539] RSP: 0018:ffffaa9387db34a0 EFLAGS: 00010002 [13936.032067] RAX: 0000000000000050 RBX: ffffaa9387db35a8 RCX: 0000000000000000 [13936.034051] RDX: 0000000000000000 RSI: ffff8a0a0211e5a8 RDI: ffff8a0a0211e5a8 [13936.035793] RBP: ffffffffa1f06ae0 R08: 0000000000000000 R09: c0000000ffff7fff [13936.038099] R10: 0000000000000001 R11: ffffaa9387db3298 R12: ffffffffa37244a8 [13936.040972] R13: 0000000000036cc0 R14: ffffffffa37244a0 R15: ffff8a08c2404f50 [13936.042784] FS: 0000000000000000(0000) GS:ffff8a0a02100000(0000) knlGS:0000000000000000 [13936.045068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.046984] CR2: 00007fddee9413e0 CR3: 00000000a1016003 CR4: 0000000000170ee0 [13936.048864] Call Trace: [13936.049464] ? show_regs.cold.9+0x22/0x2f [13936.050546] ? __warn+0xc8/0x150 [13936.051369] ? __debug_object_init.cold.5+0x35/0x15f [13936.052888] ? report_bug+0x113/0x140 [13936.053709] ? do_error_trap+0xb6/0x130 [13936.054876] ? do_invalid_op+0x46/0x60 [13936.055786] ? __debug_object_init.cold.5+0x35/0x15f [13936.056824] ? invalid_op+0x14/0x20 [13936.057693] ? __debug_object_init.cold.5+0x35/0x15f [13936.060334] ? lod_set_pool+0x260/0x260 [lod] [13936.063270] debug_object_init+0x22/0x30 [13936.064562] init_timer_key+0x28/0x120 [13936.066007] lod_ost_alloc_qos+0x790/0x1c60 [lod] [13936.067699] ? slab_post_alloc_hook+0x66/0x380 [13936.068873] ? lod_qos_prep_create+0x390/0x1bc0 [lod] [13936.070523] ? __kmalloc+0x1b4/0x4a0 [13936.071593] lod_qos_prep_create+0x134e/0x1bc0 [lod] [13936.073013] lod_prepare_create+0x204/0x460 [lod] [13936.074374] ? osd_declare_create+0x4a2/0x7a0 [osd_ldiskfs] [13936.076931] lod_declare_striped_create+0x270/0xf80 [lod] [13936.078974] ? lod_sub_declare_create+0x111/0x320 [lod] [13936.080411] lod_declare_create+0x3d4/0x9c0 [lod] [13936.082205] ? osd_xattr_get+0x274/0x910 [osd_ldiskfs] [13936.084232] mdd_declare_create_object_internal+0x107/0x4a0 [mdd] [13936.086237] ? lod_alloc_comp_entries+0x290/0x640 [lod] [13936.088201] mdd_declare_create_object.isra.25+0x55/0xc40 [mdd] [13936.090436] mdd_declare_create+0x6a/0x6c0 [mdd] [13936.091864] mdd_create+0x5bd/0x1d00 [mdd] [13936.093387] ? mdt_version_save+0xa8/0x210 [mdt] [13936.094822] mdt_reint_open+0x337c/0x3c10 [mdt] [13936.096403] ? old_init_ucred_common+0x19e/0x850 [mdt] [13936.098099] mdt_reint_rec+0x139/0x2b0 [mdt] [13936.099695] mdt_reint_internal+0x6a0/0xdc0 [mdt] [13936.100794] mdt_intent_open+0x180/0x5b0 [mdt] [13936.102526] mdt_intent_opc.constprop.43+0x153/0xfb0 [mdt] [13936.104270] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] [13936.106965] mdt_intent_policy+0x14b/0x670 [mdt] [13936.109812] ldlm_lock_enqueue+0x43c/0xcd0 [ptlrpc] [13936.112616] ? _raw_read_unlock+0x12/0x30 [13936.113891] ? cfs_hash_rw_unlock+0x11/0x30 [obdclass] [13936.115191] ldlm_handle_enqueue+0xcaf/0x2280 [ptlrpc] [13936.116852] tgt_enqueue+0xd0/0x300 [ptlrpc] [13936.118327] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [13936.119973] tgt_request_handle+0x575/0x1e70 [ptlrpc] [13936.122102] ? obd_export_timed_fini+0xc4/0xe0 [obdclass] [13936.123638] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [13936.126659] ? woken_wake_function+0x30/0x30 [13936.127964] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13936.129395] ptlrpc_main+0xce8/0x1400 [ptlrpc] [13936.130906] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [13936.132610] kthread+0x1d1/0x200 [13936.133507] ? set_kthread_struct+0x70/0x70 [13936.135072] ret_from_fork+0x1f/0x30 [13936.136562] ---[ end trace 8ff8e2d705defe30 ]--- [13936.186861] ODEBUG: object 000000007e566a9b is on stack 000000006f01246b, but NOT annotated. [13936.188954] WARNING: CPU: 3 PID: 227640 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x15f [13936.190886] Modules linked in: dm_flakey lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix serio_raw ghash_clmulni_intel libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: obdecho] [13936.201460] CPU: 3 PID: 227640 Comm: mdt00_005 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [13936.203856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [13936.205698] RIP: 0010:__debug_object_init.cold.5+0x35/0x15f [13936.206879] Code: 7e a1 48 83 05 23 61 0c 03 01 89 05 59 69 0c 03 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 93 68 99 ff 48 83 05 1b 61 0c 03 01 <0f> 0b 48 83 05 19 61 0c 03 01 48 83 05 19 61 0c 03 01 e9 3f ee ff [13936.210813] RSP: 0018:ffffaa938d8b74a0 EFLAGS: 00010006 [13936.211941] RAX: 0000000000000050 RBX: ffffaa938d8b75a8 RCX: 0000000000000000 [13936.213523] RDX: 0000000000000000 RSI: ffff8a0a0219e5a8 RDI: ffff8a0a0219e5a8 [13936.214967] RBP: ffffffffa1f06ae0 R08: 0000000000000000 R09: c0000000ffff7fff [13936.216525] R10: 0000000000000001 R11: ffffaa938d8b7298 R12: ffffffffa370bb88 [13936.217971] R13: 000000000001e3a0 R14: ffffffffa370bb80 R15: ffff8a0907d35c08 [13936.219534] FS: 0000000000000000(0000) GS:ffff8a0a02180000(0000) knlGS:0000000000000000 [13936.221155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.222401] CR2: 00005585dae9bbb8 CR3: 00000000a1016003 CR4: 0000000000170ee0 [13936.223815] Call Trace: [13936.224357] ? show_regs.cold.9+0x22/0x2f [13936.225162] ? __warn+0xc8/0x150 [13936.225777] ? __debug_object_init.cold.5+0x35/0x15f [13936.226754] ? report_bug+0x113/0x140 [13936.227545] ? do_error_trap+0xb6/0x130 [13936.228360] ? do_invalid_op+0x46/0x60 [13936.229112] ? __debug_object_init.cold.5+0x35/0x15f [13936.230184] ? invalid_op+0x14/0x20 [13936.230901] ? __debug_object_init.cold.5+0x35/0x15f [13936.231930] ? lod_set_pool+0x260/0x260 [lod] [13936.232888] debug_object_init+0x22/0x30 [13936.233753] init_timer_key+0x28/0x120 [13936.234519] lod_ost_alloc_qos+0x790/0x1c60 [lod] [13936.235592] ? slab_post_alloc_hook+0x66/0x380 [13936.236537] ? lod_qos_prep_create+0x390/0x1bc0 [lod] [13936.237616] ? __kmalloc+0x1b4/0x4a0 [13936.238400] lod_qos_prep_create+0x134e/0x1bc0 [lod] [13936.239453] lod_prepare_create+0x204/0x460 [lod] [13936.240538] ? osd_declare_create+0x4a2/0x7a0 [osd_ldiskfs] [13936.241808] lod_declare_striped_create+0x270/0xf80 [lod] [13936.242979] ? lod_sub_declare_create+0x111/0x320 [lod] [13936.244174] lod_declare_create+0x3d4/0x9c0 [lod] [13936.245238] mdd_declare_create_object_internal+0x107/0x4a0 [mdd] [13936.246600] ? lod_alloc_comp_entries+0x290/0x640 [lod] [13936.247692] mdd_declare_create_object.isra.25+0x55/0xc40 [mdd] [13936.248981] mdd_declare_create+0x6a/0x6c0 [mdd] [13936.250025] mdd_create+0x5bd/0x1d00 [mdd] [13936.250954] ? mdt_version_save+0xa8/0x210 [mdt] [13936.252115] mdt_reint_open+0x337c/0x3c10 [mdt] [13936.253117] ? old_init_ucred_common+0x19e/0x850 [mdt] [13936.254381] mdt_reint_rec+0x139/0x2b0 [mdt] [13936.255377] mdt_reint_internal+0x6a0/0xdc0 [mdt] [13936.256473] mdt_intent_open+0x180/0x5b0 [mdt] [13936.257497] mdt_intent_opc.constprop.43+0x153/0xfb0 [mdt] [13936.258704] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] [13936.259942] mdt_intent_policy+0x14b/0x670 [mdt] [13936.260948] ldlm_lock_enqueue+0x43c/0xcd0 [ptlrpc] [13936.262344] ? _raw_read_unlock+0x12/0x30 [13936.263175] ? cfs_hash_rw_unlock+0x11/0x30 [obdclass] [13936.264395] ldlm_handle_enqueue+0xcaf/0x2280 [ptlrpc] [13936.265667] tgt_enqueue+0xd0/0x300 [ptlrpc] [13936.266822] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [13936.268133] tgt_request_handle+0x575/0x1e70 [ptlrpc] [13936.269334] ? obd_export_timed_fini+0xc4/0xe0 [obdclass] [13936.270552] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [13936.272016] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13936.273226] ptlrpc_main+0xce8/0x1400 [ptlrpc] [13936.274350] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [13936.275605] kthread+0x1d1/0x200 [13936.276242] ? set_kthread_struct+0x70/0x70 [13936.277146] ret_from_fork+0x1f/0x30 [13936.277888] ---[ end trace 8ff8e2d705defe31 ]--- [13936.303448] ODEBUG: object 00000000ff768b54 is on stack 00000000953be36a, but NOT annotated. [13936.304325] ODEBUG: object 000000006d525d0b is on stack 0000000026c9b8f6, but NOT annotated. [13936.306347] WARNING: CPU: 2 PID: 220233 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x15f [13936.309010] WARNING: CPU: 0 PID: 220334 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x15f [13936.312818] Modules linked in: [13936.317264] Modules linked in: dm_flakey [13936.318521] dm_flakey [13936.318524] lustre(O) [13936.319811] lustre(O) [13936.321061] osp(O) [13936.322375] osp(O) [13936.323546] ofd(O) [13936.324067] ofd(O) [13936.325352] lod(O) [13936.326070] lod(O) [13936.326921] mdt(O) [13936.327486] mdt(O) [13936.328332] mdd(O) [13936.329027] mdd(O) [13936.329659] mgs(O) [13936.330142] mgs(O) [13936.330806] osd_ldiskfs(O) [13936.331509] osd_ldiskfs(O) [13936.332282] ldiskfs(O) [13936.333145] ldiskfs(O) [13936.333861] lquota(O) [13936.334617] lquota(O) [13936.335356] lfsck(O) [13936.336031] lfsck(O) [13936.336664] mgc(O) [13936.337290] mgc(O) [13936.337750] mdc(O) [13936.338177] mdc(O) [13936.338643] lov(O) [13936.339135] lov(O) [13936.339642] osc(O) [13936.340077] osc(O) [13936.340537] lmv(O) [13936.341166] lmv(O) [13936.341735] fid(O) [13936.342602] fid(O) [13936.343366] fld(O) [13936.343963] fld(O) [13936.344803] ptlrpc_gss(O) [13936.345211] ptlrpc_gss(O) [13936.346375] ptlrpc(O) [13936.347410] ptlrpc(O) [13936.349095] obdclass(O) [13936.350111] obdclass(O) [13936.351442] ksocklnd(O) [13936.352375] ksocklnd(O) [13936.353508] lnet(O) [13936.354284] lnet(O) libcfs(O) [13936.355456] libcfs(O) [13936.356151] zfs(O) [13936.357657] zfs(O) [13936.358317] spl(O) [13936.359223] spl(O) [13936.359696] ec(O) [13936.360622] ec(O) [13936.361135] rpcsec_gss_krb5 [13936.362199] rpcsec_gss_krb5 [13936.362679] auth_rpcgss [13936.364076] auth_rpcgss [13936.364720] nfsv4 dns_resolver [13936.365913] nfsv4 [13936.367019] intel_rapl_msr [13936.369081] dns_resolver [13936.369958] intel_rapl_common [13936.371641] intel_rapl_msr [13936.372530] sb_edac [13936.374044] intel_rapl_common [13936.375059] rapl [13936.376305] sb_edac [13936.377460] pcspkr [13936.378453] rapl [13936.379492] i2c_piix4 squashfs [13936.380701] pcspkr [13936.381370] crct10dif_pclmul crc32_pclmul [13936.382684] i2c_piix4 [13936.383235] crc32c_intel [13936.385928] squashfs [13936.386972] ata_generic [13936.388597] crct10dif_pclmul [13936.389878] ata_piix [13936.391265] crc32_pclmul [13936.393104] serio_raw [13936.394217] crc32c_intel [13936.395090] ghash_clmulni_intel [13936.396279] ata_generic [13936.396997] libata [13936.398468] ata_piix [13936.399031] dm_mirror [13936.399835] serio_raw [13936.400456] dm_region_hash [13936.401790] ghash_clmulni_intel [13936.402311] dm_log [13936.403550] libata [13936.404494] dm_mod [13936.405811] dm_mirror [13936.406559] sha512_ssse3 [13936.408006] dm_region_hash [13936.408872] sha512_generic [13936.410328] dm_log [13936.411477] [last unloaded: obdecho] [13936.412520] dm_mod [13936.413601] [13936.413606] CPU: 2 PID: 220233 Comm: mdt00_001 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [13936.415929] sha512_ssse3 [13936.417264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [13936.418178] sha512_generic [13936.422831] RIP: 0010:__debug_object_init.cold.5+0x35/0x15f [13936.423966] [last unloaded: obdecho] [13936.427159] Code: 7e a1 48 83 05 23 61 0c 03 01 89 05 59 69 0c 03 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 93 68 99 ff 48 83 05 1b 61 0c 03 01 <0f> 0b 48 83 05 19 61 0c 03 01 48 83 05 19 61 0c 03 01 e9 3f ee ff [13936.428687] [13936.431461] RSP: 0018:ffffaa938917f4a0 EFLAGS: 00010002 [13936.433568] CPU: 0 PID: 220334 Comm: mdt00_003 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 [13936.438379] [13936.438382] RAX: 0000000000000050 RBX: ffffaa938917f5a8 RCX: 0000000000000000 [13936.438934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [13936.440000] RDX: 0000000000000000 RSI: ffff8a0a0211e5a8 RDI: ffff8a0a0211e5a8 [13936.443938] RIP: 0010:__debug_object_init.cold.5+0x35/0x15f [13936.444351] RBP: ffffffffa1f06ae0 R08: 30207463656a626f R09: 203a47554245444f [13936.446915] Code: 7e a1 48 83 05 23 61 0c 03 01 89 05 59 69 0c 03 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 93 68 99 ff 48 83 05 1b 61 0c 03 01 <0f> 0b 48 83 05 19 61 0c 03 01 48 83 05 19 61 0c 03 01 e9 3f ee ff [13936.448827] R10: 30207463656a626f R11: 203a47554245444f R12: ffffffffa370b2a8 [13936.451429] RSP: 0018:ffffaa938b49f4a0 EFLAGS: 00010006 [13936.453371] R13: 000000000001dac0 R14: ffffffffa370b2a0 R15: ffff8a09d4548028 [13936.455999] [13936.459945] FS: 0000000000000000(0000) GS:ffff8a0a02100000(0000) knlGS:0000000000000000 [13936.462496] RAX: 0000000000000050 RBX: ffffaa938b49f5a8 RCX: 0000000000000000 [13936.464514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.466060] RDX: 0000000000000000 RSI: ffff8a0a0201e5a8 RDI: ffff8a0a0201e5a8 [13936.466459] CR2: 00007fddee9413e0 CR3: 00000000a1016003 CR4: 0000000000170ee0 [13936.468107] RBP: ffffffffa1f06ae0 R08: 30207463656a626f R09: 203a47554245444f [13936.469666] Call Trace: [13936.470827] R10: 30207463656a626f R11: 203a47554245444f R12: ffffffffa375ce08 [13936.472443] ? show_regs.cold.9+0x22/0x2f [13936.473855] R13: 000000000006f620 R14: ffffffffa375ce00 R15: ffff8a08d6bca898 [13936.475464] ? __warn+0xc8/0x150 [13936.476393] FS: 0000000000000000(0000) GS:ffff8a0a02000000(0000) knlGS:0000000000000000 [13936.478222] ? __debug_object_init.cold.5+0x35/0x15f [13936.479686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.481316] ? report_bug+0x113/0x140 [13936.482454] CR2: 00007fddf3dc0000 CR3: 00000000a1016003 CR4: 0000000000170ef0 [13936.484233] ? do_error_trap+0xb6/0x130 [13936.485904] Call Trace: [13936.487984] ? do_invalid_op+0x46/0x60 [13936.488791] ? show_regs.cold.9+0x22/0x2f [13936.490361] ? __debug_object_init.cold.5+0x35/0x15f [13936.491096] ? __warn+0xc8/0x150 [13936.491673] ? invalid_op+0x14/0x20 [13936.492446] ? __debug_object_init.cold.5+0x35/0x15f [13936.493385] ? __debug_object_init.cold.5+0x35/0x15f [13936.494358] ? report_bug+0x113/0x140 [13936.495226] ? lod_set_pool+0x260/0x260 [lod] [13936.495931] ? do_error_trap+0xb6/0x130 [13936.497000] debug_object_init+0x22/0x30 [13936.498005] ? do_invalid_op+0x46/0x60 [13936.499517] init_timer_key+0x28/0x120 [13936.500390] ? __debug_object_init.cold.5+0x35/0x15f [13936.501212] lod_ost_alloc_qos+0x790/0x1c60 [lod] [13936.501998] ? invalid_op+0x14/0x20 [13936.502853] ? get_page_from_freelist+0x406/0x5d0 [13936.503578] ? __debug_object_init.cold.5+0x35/0x15f [13936.504672] ? slab_post_alloc_hook+0x66/0x380 [13936.505644] ? lod_set_pool+0x260/0x260 [lod] [13936.506439] ? lod_qos_prep_create+0x390/0x1bc0 [lod] [13936.507425] debug_object_init+0x22/0x30 [13936.508497] ? __kmalloc+0x1b4/0x4a0 [13936.509377] init_timer_key+0x28/0x120 [13936.510211] lod_qos_prep_create+0x134e/0x1bc0 [lod] [13936.511213] lod_ost_alloc_qos+0x790/0x1c60 [lod] [13936.512195] lod_prepare_create+0x204/0x460 [lod] [13936.512957] ? slab_post_alloc_hook+0x66/0x380 [13936.514087] ? osd_declare_create+0x4a2/0x7a0 [osd_ldiskfs] [13936.515093] ? lod_qos_prep_create+0x390/0x1bc0 [lod] [13936.516146] lod_declare_striped_create+0x270/0xf80 [lod] [13936.517051] ? __kmalloc+0x1b4/0x4a0 [13936.517975] ? lod_sub_declare_create+0x111/0x320 [lod] [13936.519094] lod_qos_prep_create+0x134e/0x1bc0 [lod] [13936.520233] lod_declare_create+0x3d4/0x9c0 [lod] [13936.521316] lod_prepare_create+0x204/0x460 [lod] [13936.522733] mdd_declare_create_object_internal+0x107/0x4a0 [mdd] [13936.523771] ? osd_declare_create+0x4a2/0x7a0 [osd_ldiskfs] [13936.524875] ? lod_alloc_comp_entries+0x290/0x640 [lod] [13936.525844] lod_declare_striped_create+0x270/0xf80 [lod] [13936.526890] mdd_declare_create_object.isra.25+0x55/0xc40 [mdd] [13936.528106] ? lod_sub_declare_create+0x111/0x320 [lod] [13936.529403] mdd_declare_create+0x6a/0x6c0 [mdd] [13936.530444] lod_declare_create+0x3d4/0x9c0 [lod] [13936.531663] mdd_create+0x5bd/0x1d00 [mdd] [13936.532848] mdd_declare_create_object_internal+0x107/0x4a0 [mdd] [13936.534770] ? mdt_version_save+0xa8/0x210 [mdt] [13936.535670] ? lod_alloc_comp_entries+0x290/0x640 [lod] [13936.536758] mdt_reint_open+0x337c/0x3c10 [mdt] [13936.537568] mdd_declare_create_object.isra.25+0x55/0xc40 [mdd] [13936.538864] ? old_init_ucred_common+0x19e/0x850 [mdt] [13936.539770] mdd_declare_create+0x6a/0x6c0 [mdd] [13936.540951] mdt_reint_rec+0x139/0x2b0 [mdt] [13936.541879] mdd_create+0x5bd/0x1d00 [mdd] [13936.543120] mdt_reint_internal+0x6a0/0xdc0 [mdt] [13936.544141] ? mdt_version_save+0xa8/0x210 [mdt] [13936.545120] mdt_intent_open+0x180/0x5b0 [mdt] [13936.546006] mdt_reint_open+0x337c/0x3c10 [mdt] [13936.547025] mdt_intent_opc.constprop.43+0x153/0xfb0 [mdt] [13936.547986] ? old_init_ucred_common+0x19e/0x850 [mdt] [13936.549323] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] [13936.550184] mdt_reint_rec+0x139/0x2b0 [mdt] [13936.551199] mdt_intent_policy+0x14b/0x670 [mdt] [13936.552341] mdt_reint_internal+0x6a0/0xdc0 [mdt] [13936.553503] ldlm_lock_enqueue+0x43c/0xcd0 [ptlrpc] [13936.554543] mdt_intent_open+0x180/0x5b0 [mdt] [13936.555701] ? _raw_read_unlock+0x12/0x30 [13936.556627] mdt_intent_opc.constprop.43+0x153/0xfb0 [mdt] [13936.558146] ? cfs_hash_rw_unlock+0x11/0x30 [obdclass] [13936.559147] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] [13936.560119] ldlm_handle_enqueue+0xcaf/0x2280 [ptlrpc] [13936.560940] mdt_intent_policy+0x14b/0x670 [mdt] [13936.562181] tgt_enqueue+0xd0/0x300 [ptlrpc] [13936.563219] ldlm_lock_enqueue+0x43c/0xcd0 [ptlrpc] [13936.564386] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [13936.565374] ? _raw_read_unlock+0x12/0x30 [13936.566546] tgt_request_handle+0x575/0x1e70 [ptlrpc] [13936.567405] ? cfs_hash_rw_unlock+0x11/0x30 [obdclass] [13936.569362] ? obd_export_timed_fini+0xc4/0xe0 [obdclass] [13936.570379] ldlm_handle_enqueue+0xcaf/0x2280 [ptlrpc] [13936.571253] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [13936.572278] tgt_enqueue+0xd0/0x300 [ptlrpc] [13936.573413] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13936.574430] tgt_handle_request0+0x137/0xaf0 [ptlrpc] [13936.575548] ptlrpc_main+0xce8/0x1400 [ptlrpc] [13936.576740] tgt_request_handle+0x575/0x1e70 [ptlrpc] [13936.577663] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [13936.578795] ? obd_export_timed_fini+0xc4/0xe0 [obdclass] [13936.579899] kthread+0x1d1/0x200 [13936.580848] ptlrpc_server_handle_request+0x443/0x13b0 [ptlrpc] [13936.581998] ? set_kthread_struct+0x70/0x70 [13936.582977] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [13936.584222] ret_from_fork+0x1f/0x30 [13936.584926] ptlrpc_main+0xce8/0x1400 [ptlrpc] [13936.586337] ---[ end trace 8ff8e2d705defe32 ]--- [13936.587154] ? ptlrpc_wait_event+0x690/0x690 [ptlrpc] [13936.591918] kthread+0x1d1/0x200 [13936.592559] ? set_kthread_struct+0x70/0x70 [13936.593431] ret_from_fork+0x1f/0x30 [13936.594156] ---[ end trace 8ff8e2d705defe33 ]--- [13968.829408] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 00:17:32 (1778300252) [14218.452370] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 00:21:42 (1778300502) [14219.112614] Lustre: *** cfs_fail_loc=148, val=0*** [14222.299997] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 00:21:46 (1778300506) [14227.762391] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 00:21:51 (1778300511) [14230.993121] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 00:21:55 (1778300515) [14242.042245] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 00:22:06 (1778300526) [14244.813725] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 00:22:08 (1778300528) [14247.529783] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 00:22:11 (1778300531) [14249.503596] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 00:22:13 (1778300533) [14252.654222] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 00:22:16 (1778300536) [14258.837254] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 00:22:22 (1778300542) [14261.668157] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 00:22:25 (1778300545) [14263.772653] Lustre: DEBUG MARKER: == sanity test 190a: check lfs project -p works with project name ========================================================== 00:22:27 (1778300547) [14265.864021] Lustre: DEBUG MARKER: == sanity test 190b: check lfs find --project works with project name ========================================================== 00:22:29 (1778300549) [14273.987297] Lustre: DEBUG MARKER: == sanity test 190c: check lfs project -p works with u:USERNAME ========================================================== 00:22:38 (1778300558) [14278.366937] Lustre: DEBUG MARKER: == sanity test complete, duration 13959 sec ============== 00:22:42 (1778300562) [14278.905202] Lustre: DEBUG MARKER: === sanity: start cleanup 00:22:42 (1778300562) === [14293.721346] Lustre: DEBUG MARKER: === sanity: finish cleanup 00:22:57 (1778300577) === [14295.519786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14295.521950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14295.525053] Lustre: Skipped 1 previous similar message [14295.526572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14301.069512] Lustre: server umount lustre-MDT0000 complete [14302.176774] LustreError: 259822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.181101] LustreError: 259822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [14303.508069] LustreError: 182376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778300587 with bad export cookie 5489915982265599390 [14303.511311] LustreError: 182376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14303.512723] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14303.649280] Lustre: server umount lustre-MDT0001 complete [14315.547882] Lustre: server umount lustre-OST0000 complete [14327.291880] Lustre: server umount lustre-OST0001 complete [14332.492384] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [14333.293904] Key type lgssc unregistered [14333.416543] LNet: 273688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14333.420371] LNetError: 273688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14333.427417] LNet: Removed LNI 192.168.204.106@tcp [14333.695092] Key type .llcrypt unregistered [14333.696122] Key type ._llcrypt unregistered